The present castle was constructed in the 1560s in the then Danish province of Scania by Danish riksråd Sten Rosensparre, though certain parts of the castle are presumed to be older.

In reality, however, it was bought by the funds of his rich wife, Beata Elisabet von Königsmarck who, as a married woman and thereby a minor, could not formally be listed as its buyer and owner until after she was widowed in 1692.

In practice, Beata Elisabet von Königsmarck managed the estate alone from 1661 until her death in 1723, when it was inherited by the grandson of her daughter, count Erik Brahe.

His son, Magnus Fredrik Brahe, sold Skarholt to Charles XIV John of Sweden in 1826.

Skarholt was sold to baron Jules von Schwerin by Oscar I of Sweden and is still privately owned.