Following the Norman Conquest, Richard Fitz Gilbert was granted land in Kent to guard the crossing of the River Medway.

In 1088, the de Clare family (descendants of Fitz Gilbert) rebelled against King William II.

[1] The twin-towered gatehouse was built by Richard de Clare, third Earl of Hertford or his son Gilbert.

[3] The site was purchased by the local council in 1900, who now use the mansion as offices, and who made the grounds a public park.

The castle is also the start of a 6-mile cycle ride to Penshurst Place called The Tudor Trail.