After the death of Conrad II of Schlüsselberg, the castle and town of Waischenfeld went in 1348 into the hands of the Bishopric of Bamberg.

In 1430 the castle was badly damaged during the Hussite Wars; it was then re-enfeoffed by the bishopric to the nobility between 1438 and 1562.

During the conflict the castle was captured three times by the margrave's troops; after being plundered it was razed on 7 June 1553.

The tower known as the Steinerner Beutel ("Stone Bag") is the symbol of the town of Waischenfeld.

The Romanesque round tower is roughly 13 metres high and stands on a limestone crag.