The estate traces its history back to the 13th century but the current asymmetrical complex of single-storey, white-washed buildings surrounding a central courtyard was constructed for Hans Didrik Brinck-Seidelin in 1747 with the exception of the remains of a tower built by Johan Friis in 1555.

The ravages and looting of Swedish troops during the Dano-Swedish War economically ruined Thott.

As compensation, Christian V granted him tax exemption, contributed with building materials for the rebuilding of the house and appointed him as court physician.

Ole Bornamann (1673-1725), a district judge of Zealand and Møn and the owner of Nørager at Kalundborg, purchased the estate in 1709.

In 1725, Lars Benzon sold Hagestedgaard to civil servant and poet Willum Worm (1698-1737).

In 1752, Brinck-Seidelin founded Stamhuset Hagested (a family trust) from the estates of Hagestedgaard, Holbæk Ladegård and Eriksholm.

His son, Christian Ludvig Castenschiold sold Hagestedgaard to his cousin Casper Holten Grevencop-Castenschiold in 1825, due to economic difficulties.

One of the daughters, Severine Jacobine Grevencop-Castenschiold, married the prominent archaeologist Jens Jacob Asmussen Worsaae.

Her only child, Esther, Baroness Lerche, who had been born on the estate in 1909 (died 1986), married Frants Hvass (1896-1982), a diplomat, on 17 September 1949.

The buildings are built with timber framing on foundations of fieldstone and have roofs clad with winged, red tile.

[3] The main wing is located on the west side of the courtyard and incorporates the remains of a tower and a barrel-vaulted cellar built by Johan Friis in 1555.

Two small pavilions with half-hip roofs are located on the north side of the central courtyard.