It is an excellent example of palace architecture typical for the alpine parts of the Holy Roman Empire during the late Gothic and early Renaissance period.
Within 8 years, starting in 1500 a three-storey four-wing complex with courtyard and stair tower, portals and arcades, columns, window sills and fireplaces in Hagauer marble was built.
The subsequent owner, Augsburg knight Georg von Ilsung finished the north wing and also the inner yards striking façade painting originate from this era.
The wealthy Augsburg merchant family Fugger were also deeply involved in the local copper and silver mining businesses and inherited the castle in 1589.
During the 17th and 18th century the castle underwent numerous changes of ownership and was owned by the noble families of Stauber-Imhof, Von der Halden and Josef Ignaz Reichsfreiherren (Barons of the Empire) of Tannenberg.