The main building was built between in 1559-1575 in Dutch Renaissance style.

Peter Rud, a friend of king Frederick II, died in 1559.

His widow Grethe Bruske constructed the current main building in c. 1559-75.

Severin Løvenskiold had inherited Birkholm Manor and Aggersvold as well as several iron works in Norway.

The Barony of Løvenborg was dissolved in 1921 existed until 1921 as a result of the lens release (lensafløsningen) of 1919.