By 1145 the free knights of Reifenberg had moved from Ettling near Ingolstadt.

In 1160 Emperor Frederick I confirmed the estate of Reifenberg Castle to Bishop Eberhard II of Bamberg and in 1185 Pope Lucius III confirmed the estate to the Bishopric of Bamberg.

In 1430, during the Hussite Wars, the castle was damaged, but rebuilt in stages up to 1446.

In 1525, at the time of the Palatine Peasants' War, the castle had probably already become derelict.

From 1607 to 1613 the Church of St. Nicholas (Vexier Chapel) was rebuilt and, in 1788, was given a baroque tower and hence its present appearance.