It is uncertain who founded the castle: if it was early it could have been built on the instruction of William the Conqueror as he subdued south-west England or if it was a later construction it could have been built for Juhel (Joel) of Totnes, who held the castle in the early 12th century.
In the 12th century, stone buildings were erected on top of the motte, possibly during Henry de Tracy's tenure.
[5][6] By the time of the death of the last Henry de Tracey in 1274, the castle was beginning to decay.
"[7] Adam Wyat recorded that part of the castle walls blew down in a storm in 1601.
[9][10] Since 1950, the castle has been designated a scheduled monument, which is intended to protect important archaeological sites.