Built by Norman knight Adam de Rupe in the second half of the 12th century,[1] probably on the site of an earlier wooden structure.

Their daughter Lucy was born in the castle, and later became a courtesan of Charles II, and bore him an acknowledged son James, 1st Duke of Monmouth.

The castle changed hands twice, before the Walter family fled when it was captured for a second time by the Parliamentary forces and subsequently burned in 1644.

It lay dormant and unclaimed until the return of Duke of Monmouth, who led a rebellion and was captured at the Battle of Sedgemoor, and then beheaded on 15 July 1685.

The castle remained decaying until 1900, when purchased by John Philipps, 1st Viscount St Davids, who restored it with a steel frame and new concrete floors to the designs of D. E. Thomas of Haverfordwest.