[2] The fortress served as an administrative seat in the Upper Carinthian estates held by the Counts of Ortenburg and Celje; it later was the residence of the Graben and Khevenhüller dynasties; ministeriales of the Austrian House of Habsburg.

The castle was probably erected in the 12th century, as one Witemarus de Sumereke was already mentioned in an 1187 deed issued at Neustift Abbey in Tyrol.

The Lords of Sommeregg then served as ministeriales of Count Otto II of Ortenburg, who ruled over extended estates in Upper Carinthia, rivalling with the House of Gorizia and the Salzburg archbishops.

In the 14th century, the Lords of Sommeregg achieved the knightly status of Ortenburg burgraves and castellans with comprehensive administrative and military responsibilities in the lordship of the manor.

After a long dispute with Count John II of Gorizia, the former Ortenburg possessions fell to the Habsburg emperor Frederick III, who confirmed the feudal rights of the Graben family.