The house was built for Sir Humphrey Monoux, 1st Baronet on land he had acquired from Lord Carlisle around 1660.

The majority of the estate was inherited by the eldest daughter Mary, the widow of Sir John Payne, Bt who went on to remarry Francis Buckworth.

She remained resident in the house until 1850 after which her grandson Sir Coventry Payne, Bt moved in.

By this time the family fortunes, based on West Indian plantations, had dwindled and the property was heavily mortgaged.

Philip was obliged to sell it to his daughter Sybil Harriet Doyne-Ditmas in 1923, who resold it in a public sale in 1927.