As one of the earliest Gothic Revival houses, the building shaped the course of British architectural history in an indirect way: when the Houses of Parliament were to be rebuilt after the fire in 1834, Hanbury-Tracy headed the jury to the competition, and the architect of the winning design, Charles Barry, obviously adapted his entry to the taste exemplified in Toddington.

[2] The family owned the house until 1893 when Charles Hanbury-Tracy, 4th Baron Sudeley, and his writer wife Ada had to sell due to bankruptcy.

The last private owner, Isabel Andrews, whose husband had bought the estate in 1901, died in 1935 and it stood empty until September 1939, when it was purchased by the National Union of Teachers, who had moved out of London to avoid air raids.

Following Dunkirk a tented encampment was erected in the grounds and temporarily occupied by men evacuated from the beaches.

In 2004, following the school's closure, planning permission to convert it into a hotel was denied after the scheme had attracted considerable local opposition.