It was said to have been built for King Henry II, although archaeological excavations carried out in the 1920s suggest that it stands on the foundations of a much older Anglo-Saxon fortification, possibly dating from the seventh century.

In June 1320, Chilham Castle was the venue for a reception hosted by Bartholomew de Badlesmere for Edward II and his entourage when they were travelling through Dover en route for France.

Plans of Chilham showing some of the substantial changes made to the building by David Brandon for Charles Hardy in 1862 and by Herbert Baker for mining magnate Sir Edmund and Lady Davis in the early 1920s are conserved in the Victoria and Albert Museum.

[9] Chilham Castle was owned by the financier and UKIP activist Stuart Wheeler, who lived there with his three daughters, Sarah, Jacquetta, and Charlotte until his death in 2020.

[citation needed] In 1965, Chilham Castle was used for part of the filming of The Amorous Adventures of Moll Flanders starring Kim Novak, Leo McKern and Angela Lansbury.

In 1989, it also featured in the first episode of the ITV adventure game show Interceptor produced by Chatsworth Television who were responsible for the earlier Treasure Hunt series.