[1] Nevertheless the County of Werdenfels, maintained territorial claims up to the outskirts of Seefeld, citing the bishopric's borders of 1060 and later, unilateral, boundary records.

At the outbreak of the War of the Tyrolean Succession in 1335 the castle, as an important border fortification, was further fortified and a tax, the steura nova raised to pay for it.

In spite of these precautions, the castle was conquered in 1365 and 1368 by Bavarian troops, but was recaptured shortly afterwards by a Tyrolean contingent under the leadership of Petermann of Schenna, Burgrave of Tyrol.

He appears not to have been there long because as early as 1460 another superintendent, Burghard von Hausen, was in charge and he extended the castle at the behest of Sigismund.

On 20 October 1500, Sigismund's successor, Maximilian I and Prince-Bishop Philip of Freising ratified the treaty agreed the year before which saw the border of Tyrol moved northwards to a kilometre in front of Scharnitz.

These ended under the next incumbent, Johann Gwarientis (from 1569), because Archduke Ferdinand of Tyrol underwrote the castle in 1586 with all the estate of the parish of Seefeld.

In 1633, after Tyrol had received permission from the Prince-Bishopric of Freising to build a fortification in the narrow section of the valley at Scharnitz, the subsequent Porta Claudia, Schlossberg Castle lost its importance and rapidly fell into ruins.

The road through the Scharenz runs past the castle site; a route that for centuries has linked Mittenwald and the Inn valley.

The main building was surrounded by a medieval Zwinger with walls 1.2 to 3.6 metres (3.9 to 11.8 ft) thick, the southwest corner of which was fortified with a roundel (Rondell).

A wall or letzi blocking access along the gorge (Klausenmauer, Wegsperre) ran from the east side of the castle to the street below and from there up the opposite hillside where it ended at the rocks.