On this site, across the road from St Mary's Church, stood a three-bay Jacobean manor house, dating from 1660.
The estate was bought in 1828 by Joseph Neeld, a London lawyer who had inherited a substantial sum, and Grittleton became his country seat.
Joseph Neeld (1789–1856), from Hendon, Middlesex, who commissioned the building of Grittleton House, was a barrister and Member of Parliament.
[4] In 1827 he inherited an enormous fortune from his great uncle Philip Rundell who was a London silversmith and had been appointed as Goldsmith and Jeweller to the King.
[5] Neeld was a collector of British art, and – according to the historian John Britton – by about 1830 he had numerous works by Chantrey, Gibson, Baily, Wyatt, Papworth, Constable, Gainsborough, Etty, Roberts, Stanfield, Wilson, Ward, West, and many others.
He died in 1941, and as he had no heirs the property was inherited by a descendant of Joseph Neeld's illegitimate daughter Ann Maria, wife of Lieutenant Colonel Inigo William Jones.
Lionel William Neeld Inigo-Jones (1885–1956) and his successors remained at their main home, Kelston Park near Bath, and rented Grittleton House to various tenants.
[12] Thomson also designed Fosse Lodge, in the north of the estate on the Fosse Way, with a tall, octagonal tower (1835);[13] and Malmesbury Lodge in Grittleton village (c. 1840), its octagonal spirelet with bell-stage noted as "remarkable" in the building's National Heritage List entry.