It stands on a hill alongside the Drava River overlooking the town, and it is a prominent landmark.
In 1656, due to financial hardship, the Jesuits sold the castle to the Holy Roman Field Marshal Walter Leslie, a Scot by birth.
General Walter Leslie turned it into a rural residence and the seat of a landed estate.
After their extinction, the castle was bought in 1873 by Countess Therese von Herberstein (1822-1895), eldest daughter of Joseph Franz, Prince of Dietrichstein.
In the years after 1945 all the buildings on the hill inside the castle were nationalized and converted into a museum of regional history.