The castle was erected as a 12th-century manor house and fortified in 1377, giving it a rhomboid shaped stonework enclosure with high curtain walls, internal towers in each corner, a hall and a gateway.
[1] The walls, gateway and two of the towers remain as a Grade I listed building[1] and are now in use as a privately owned hotel.
In 1377, at the beginning of the Hundred Years War, there was a concern that the south coast of England would be raided by the French, so the bishop applied for, and received, permission to fortify the site and add the gatehouse and oubliette.
[4] In the English Civil War the Royalist tenant caused Oliver Cromwell to order General William Waller to attack the castle, which resulted in the loss of 20 feet from the height of the walls and the destruction of the Great Hall.
[4] In 1872, the Castle was sold to Robert Curzon, 15th Lord Zouche, who used it as a hunting lodge before selling it to Henry Fitzalan-Howard, 15th Duke of Norfolk in 1893.