The castle's name, "Witilinesbac", is however already mentioned as the place of origin of Otto IV in a document by Henry V dating from 1115.

According to local tradition, the castle was destroyed in 1209 after Count Otto of Wittelsbach murdered King Philip of Swabia, and it was not rebuilt.

In the 15th century, the Gothic church Beatae Mariae Virginis was built on the former castle site.

In memory of the castle, parts of the district Aichach-Friedberg are today called Wittelsbacher country.

In 1838 Duke Maximilian Joseph in Bavaria, the father of Empress Elisabeth of Austria, acquired a castle in the nearby village of Unterwittelsbach (Lower Wittelsbach), built in 1537 on the site of another, older castle dating from the Middle Ages; it remained in the possession of the junior ducal branch (named Dukes in Bavaria) of the royal House of Wittelsbach until 1955 and now houses a museum devoted to Empress Elisabeth.