The main building was built between in 1559-1575 in Dutch Renaissance style.
Peter Rud, a friend of king Frederick II, died in 1559.
His widow Grethe Bruske constructed the current main building in c. 1559-75.
Severin Løvenskiold had inherited Birkholm Manor and Aggersvold as well as several iron works in Norway.
The Barony of Løvenborg was dissolved in 1921 existed until 1921 as a result of the lens release (lensafløsningen) of 1919.