Virtually nothing is known about the first owners of the castle, although they were probably knights in the service of the Counts of Zähringen and they may have used the name von Attinghausen.

The Freiherr von Schweinsberg first appears in documents in the 13th century from Wartenstein Castle in the Bernese Emmental.

By 1300, there were two branches: one under Werner II who held the lands in Uri, and another under Diethelm I in the Berner Oberland.

The Uri branch of the family is traditionally believed to have been critical supporters of the Three Forest Cantons and the early Swiss Confederacy.

In 1353 Johann received a fief that included the Imperial customs post at the Castle of Rudenz and he became the Rector over Valais.

Through donations from the public they raised enough money to thoroughly investigate, document and repair the castle ruins.