Prior to the Norman Conquest there was an abbey at Ascott, that had been given by a royal to a Benedictine convent in Angiers.

In 1554 William Dormer entertained Princess Elizabeth at the house, when she was on the road to London under arrest as a Protestant because her sister Mary had just taken the throne.

The house fell into decay following the death without heirs of Charles Dormer, 2nd Earl of Carnarvon in 1709.

In the late 19th century members of the Rothschild banking family began to acquire estates in the area, including Ascott.

In 1873 a farm house in the parish known as Ascott Hall was bought by Baron Mayer de Rothschild he gave it to his nephew Leopold de Rothschild who employed the architect George Devey to enlarge the property into a substantial country house.