Together with their lordship of the castles of Neuenbürg, Calw, Fautsberg, Wildberg and Nagold, they were able to maintain their influence over the newly acquired territories in the northeastern Black Forest.

In 1342, Paul of Giltlingen sold the fief of the castle for 1,530 pounds of hellers to Count Götz of Tübingen.

Due to the poor financial situation of the House of Württemberg the fief of the Barony of Zavelstein was constantly extended.

It was not until 1468 that Burkhard von Ehingen was given an order to settle the income of the castle and town of Zavelstein for 2,600 guilders.

Duke Christopher of Württemberg redeemed the pledge in 1552 from Hans of Ehingen In 1554, Jordan of Braitenbach on Roßnitz was enfeoffed with the castle.

After the baron died of the plague in 1635 in Stuttgart, his son James Frederick Buwinghausen took possession of the property.