During The Anarchy Robert was exiled by King Stephen and the castle given to Ralph Lovel who then sided with Robert, 1st Earl of Gloucester against the king.

[3] In 1143 Stephen lost control of the West Country after the Battle of Wilton.

The Lovels later achieved the return of the castle, and their descendants were lords of the manor until the 14th century.

Around that time a manor house was built on or adjacent to the site of the castle,[4] possibly by Baron Zouche.

[2] The site was excavated in 1890 and demonstrated the foundations of a 24-metre square stone keep and an inner and outer bailey.