John de Paveley died without an heir, so his lands passed through marriage to John de Sancto Laudo, a knight who gained recognition for his actions at Acre during the Crusades.

[1] At the start of the 14th century, a keep was built on the site as part of a wider programme of work by the St Loe family, creating a rectangular, courtyard castle with four corner towers, protected by a ditch on three sides.

[2][3] In 1375 the site was inherited by Elizabeth, the last of the St Loe family, who married William, Baron Botreaux.

[4] Sir Walter Hungerford built a new gatehouse at the castle during the first half of the 15th century, which features turrets, machicolations and an early gunport.

[5] In the 19th century, the castle and the surrounding park were heavily landscaped to produce the current country house and gardens; only the renovated keep and the gatehouse survive intact, although a nearby mound marks one of the corner towers of the 14th-century castle.