The castle, mostly built by Henry of Blois in the 12th century, is now a ruin, except for its fifteenth-century chapel, which is now part of the bishop's current residence, Wolvesey Palace.

Giffard's successor, Henry of Blois, brother of King Stephen added a second hall to the west between 1135 and 1138.

[3] In June 1216, Oliver fitz Regis defended the buildings when they were besieged by Prince Louis of France during the First Barons' War.

[7] The ruins are located next to the existing bishop's palace,[5] and are currently owned and maintained by English Heritage.

[7][8] A fair amount of the curtain wall remains, but nearly all the inner arrangements are gone, though it is possible to make out the hall, in which there is a good round arch and one surviving Norman window.