[1] Jörgenberg Castle was built in the 8th century as a fortified church, on lands donated by the Frankish kings.

At the beginning of the 9th century it was called ecclesia sancti Georgii in Castello or St. George's Church in a Castle.

[2] In the 14th century the castle appears in the historical record in the possession of the Freiherren von Fry(i)berg, who were vassals of the powerful Vaz family.

However, a few years later the last member of the Vaz family, Donat, died leaving the Frybergs without a powerful patron.

To avoid losing their estates, they sold Siat and Jörgenberg to the Dukes of Austria and then received them back as a fief.

In 1342 the last male heir of the Fryberg family, Reinher, died and the Austrians gave the castle to the Counts of Werdenberg.

The Freiherr von Rhäzüns demanded the castles based on his claim from 1333 and began attacking the Austrians in the region.

[4] Beginning around 1351 the Rhäzüns family rebuilt the burned tower and added a palas to the north-west corner.

The local farmers and villagers disliked the foreign Swabian Zollerns and refused to pay them homage and often rose up in rebellion.