Although thus named, the Van Duvenvoirdes properly formed part of the House of Wassenaer, an ancient noble family which has played an important role in Dutch history.

Legend holds that both stones come from Brittenburg, the presumed Roman ruins that resurfaced above water off the coast of Katwijk in 1520.

The last private owner of the castle was Jonkvrouwe Ludolphine Henriette, Baroness Schimmelpenninck van der Oye (1891-1965).

The terrace constructed around 1844 has been kept, and the opening of previously sealed windows to allow more light into the living room has taken away some symmetry.

However, the central and north wings of the house have not been turned into the usual kind of museum and instead have been made to look as though they are still lived in, giving the visitor the impression of going back in time and walking around a house from which its 18th-century owners are only temporarily absent.