The castle occupies a commanding position atop a large hill above the town, and guards the approach to three valleys.

[2] The first castle on this site is believed to have been built by the Breton Juhel of Totnes (also known as Judhael) who was one of William the Conqueror's lieutenants.

Following this it came into the possession of William de Braose, 3rd Lord of Bramber, who is probably responsible for constructing the first stone shell keep and walls on the site.

As a result, the castle was re-fortified with a new shell keep, using Devonian limestone and red sandstone, a stable was constructed and a constable appointed.

Subsequently, in 1326, the de la Zouch clan assumed ownership of the castle and undertook extensive rebuilding, transforming its structure.