The Moors built a large fortified castle or alcazaba in the early 8th century to defend the area against the Christian kings of Leon and Castile.

In 1124, the castle was retaken by Bernard of Agen allowing Simón Girón de Cisneros to build an episcopal palace there.

In 1298, the castle was taken by the troops of Alfonso de la Cerda during the battle against Ferdinand IV of Castile, the boy king, but was defended by the bishop's vassals.

[3] In the late 18th century, Bishop Juan Díaz de la Guerra changed the appearance of the castle from a fortress into an episcopal palace with additional windows, balconies and stables.

However, in 1808 during the War of Spanish Independence the castle was taken by the French who seriously damaged it and looted all its riches before it was recaptured by El Empecinado.