The construction of the fortress was begun in the late 11th century by one noble Adalbert from Bavaria, a ministerialis of the Bishops of Freising, who then held large possessions in the Upper Carinthian Lurngau around the former Roman city of Teurnia.

In 1192 Count Otto II of Ortenburg and his brother Hermann founded the settlement of Spittal below the castle.

Damaged by the 1348 Friuli earthquake, however, the significance of the comital castle diminished in favour of the nearby Spittal residence.

During the Carinthian Peasant Revolt of 1478, one of the leading insurgents, Peter Wunderlich, was imprisoned at Ortenburg Castle, before he was executed in nearby Lendorf.

In 1524 the comital title passed to Gabriel von Salamanca, a favourite of the Habsburg archduke Ferdinand I, who had a lavish new residence, present-day Porcia Castle, built in Spittal from about 1530 onwards.