The house was gradually enlarged and extended during the 19th century and finally demolished in the 20th, apart from a prospect tower and ancillary buildings.

The castle passed to T E Foster MacGeagh and he sold it in 1919 to Henry Thomas Pearson, whose family occupied it until 1946.

[3] The unoccupied castle changed hands several times after the Pearsons left, and was demolished in 1951,[2] except for the servants' quarters, several stables and the Coach House, which was saved by the painter Bernard Hailstone.

[4] Now the entrance gateway and lodges of the Castle still stand - a heavy Gothic presence on the street - as does the Stable Court with two turreted pavilions, which are all in private ownerships, and new homes have been built in the grounds.

In July 2006, Tonbridge and Malling borough council announced that it would issue a compulsory purchase order (CPO) on the tower in an effort to save it.

[7] This CPO was confirmed in March 2008 by the Secretary of State for Culture, Media and Sport, with plans for the council to take possession of the Tower and transfer it to the Vivat Trust in late 2009, so that the necessary repair and restoration work can be undertaken; plans included short-term holiday accommodation, with a separate exhibition centre on part of the ground floor.

Restoration of the tower, including the replacement of the lantern commenced in February 2011, with completion then scheduled for September 2012.

After restoration, the tower offered holiday accommodation,[10] with public exhibition space on the ground floor.