[2][3] The Castle is sited on a small isolated knoll of Leighland Slates of the Devonian series, about 390 feet (119 m) high.

[5] Alfred of Spain (who was actually from Épaignes in Normandy) is the Norman Lord of Stowey as recorded in the Domesday Book of 1086; the building of the castle is normally attributed to him or to his daughter Isabel.

The mounds around the top of the motte are assumed to be spoil heaps from these excavations to clear the foundations of the keep, although it has also been suggested that they may be the remains of towers.

[7] The blue lias rubble foundations are the only visible structural remains of the castle, which stood on the conical motte surrounded by a ditch approximately 820 feet (250 m) in circumference.

[8] The castle was destroyed in the 15th century, which may have been as a penalty for the local Lord Audley's involvement in the Second Cornish Uprising of 1497 led by Perkin Warbeck.