It was briefly owned by John and Robert Vicary but the site stood idle until acquired by paper manufacturer James Crompton in 1873.

During the following 90 years the house was variously occupied by Charles Ichabod Wright, Mr James Peck, Sir John Edwards-Moss, Frederick James Lund, Thomas John Crossman, Stockwell College of Education (evacuated from London during World War II), and the Holiday Fellowship (who gave the estate its current name).

Brunel Manor was until 2021 operated by The Woodlands House of Prayer Trust as a Christian holiday, retreat and conference centre.

A hall in matching style was later added to the east; available dates for this addition are vague, quoted by Brunel Manor's own information document as being between 1907 and 1923.

This means that many original features are now hidden behind plasterboard, corridors divided by modern fire doors or internal views seen through wired glass.