The castle of Kalø was successful in its original purpose, but already in 1320 the new king Christoffer II, was forced by the Danish nobility in a coronation charter to tear it down, along with most of the Crown's fortresses in Jutland.

From the 15th century and onwards, Kalø had a more peaceful role, as the local administrative center and state prison.

The buildings had fallen into decay under the Swedish occupation during the wars between Sweden and Denmark in the years 1643 to 1645.

In 1662, Ulrik Frederik Gyldenløve, Count of Laurvig, decided to tear down the abandoned ruin when he received it from his father, Frederick III.

The restoration proceeded through World War II, when the German marines used Kalø as a target for gunboat shooting practice.