Louis II, Duke of Bavaria acquired at that time the possession of the fortress from Ulrich Vellenberg, a ministeriales of the Counts of Andechs.
The present building dates mainly from the late 15th century, when the castle was renovated to celebrate the wedding of Albert IV, Duke of Bavaria with Kunigunde of Austria, a daughter of emperor Frederick III.
At this conference, the two co-governing dukes, William IV and Ludwig X agreed that Bavaria should continue to belong to the "old faith", but that the church should be reformed.
This decision is today understood as the beginning of the Counter-Reformation in Bavaria, the empire and also the Habsburg lands and had effects that should shape the history of Europe over the next centuries.
A project for a luxury condominium was prevented by a citizens' initiative which finally led to the purchase of the castle by the Free State of Bavaria in 1976.