During this early stage of its history it was a strongly fortified house with a ring wall, double moats and draw bridges.

[1] After that it passed through the Brahe, Rosenkrandz and Krabbe families before it was bought by Queen Consort Charlotte Amalie in 1694 as part of a larger acquisition of land in the area.

[2] The next owner was Prince Charles of Denmark, Charlotte Amalie's youngest son, who resided at Jægerspris Castle which he had been given by his brother, Frederick IV, who had become king in 1699.

When Frederick IV had Anne Sophie Reventlow, his spouse by bigamy, crowned as his queen in 1721, it led to a breach between the two brothers, and Prince Charles and their sister, Princess Sophia Hedwig, showed their disapproval by turning their backs on the Court in Copenhagen and taking up residence at Vemmetofte.

In 1882 and 1883, the architect Theodor Zeltner carried out a rather rough renovation to a Historicist style which resulted in an unfortunate attempt to recreate a Renaissance castle.