A tower house, which once stood on the site and was demolished in 1780, was described in a survey of 1715 as "a most magnificent freestone edifice in a square figure, flat roofed and embattled" and with "a handsome court and gateway on the front".

The Hall which stands on the site today was built in 1782 by Newcastle architect, William Newton.

The 2nd Earl Grey employed George Wyatt in 1809 to enlarge the house by moving the entrance to the north side, filling out the front hall and the two quadrants linking the house to its wings, and building the first terrace on the south side.

It was rebuilt in 1928 to designs by Sir Herbert Baker and George Reavell,[3] who altered the north façade by introducing a portico above the front hall, in order to make the house smaller with an open well in the middle, with a rotunda linking the front and back on the ground floor.

In 1973 his grandson, Charles Baring, 2nd Baron Howick of Glendale, converted the west wing into a home, where he and his family now live.