The property is situated at the site of the former castle 'Heerlijkheid Heemstede', or 'Huis te Heemstede', at a strategic position on mouth of the Spaarne river on the Haarlem lake (since 1853 pumped dry and called the Haarlemmermeer polder).

The castle was first built in 1280, by Dirk van Hoylede from the region of Vlaardingen.

Built, burned and rebuilt over the centuries, it was last torn down in 1810, after years of neglect.

The monumental gatekeeper's house 'Nederhuys', built in 1630, remains intact as well as the foundations from the Middle Ages.

The most famous owner of the castle was Adriaan Pauw, who bought it in 1620.