The new lords of the castle (Burgherren) were hostile towards their neighbours, the counts of Stolberg, and raided their estates.
In 1320, after a legal decision in his favour, Count Henry of Stolberg took the law into his own hands and captured Wolfsberg Castle.
In order to protect the estate from the claims of the House of Anhalt, he transferred it as a fief to Bishop Albert of Halberstadt and was officially enfeoffed with it on 18 December 1325 along with Erichsberg Castle.
The castle was damaged during the Peasants' War of 1525, but even prior to that it had become run down, because in 1511 repairs to its roof and walls were carried out at the behest of the comital Vogt of Stolberg, Volkmar of Morungen.
Wolfsberg was enfeoffed several times even in the early modern period, but finally fell into ruins in the 18th century for lack of maintenance.