In the mid-17th century, the estate belonged to Hans Rostgaard, a civil servant who was active in the combat against invading Swedish troops.

His bibliognostic son Frederik Rostgaard withdrew to Krogerup when he was banished from the Royal Court in Copenhagen in 1724 after accusations of corruption.

Constantin Brun, a successful merchant and royal administrator of trade in the Danish West Indies, bought the property in 1812 for his sons who were farmers and politicians .

By then the estate was deep in debt, and when the last Brun to own it died, none of the heirs wished to inherit the property.

When the Nazi occupants took control of the Danish police force in 1944, the HIPO Corps took over and ravaged the building.