[3] The park surrounds High Elms Golf Course, and has extensive woodland, chiefly oak and beech, chalk meadows and formal gardens.

The history of the High Elms estate can be traced back to the Norman Conquest, when it was given by William the Conqueror to his half-brother, Odo, bishop of Bayeux.

[5] In 1809, a wealthy London banker and Member of Parliament, John William Lubbock (2nd Baronet),[6] bought the 260 acres now known as the High Elms Estate as a country residence.

[7] In 1840 it was inherited by his son, the astronomer and banker Sir John Lubbock, 3rd Baronet,[8][9] who built a grand new mansion in the Italian style.

[10] When Charles Darwin moved in 1842 into the nearby Down House on the other side of the village of Downe, he had told his sister "The great Astronomer Sir J. Lubbock is owner of 3000 acres here, & is building a grand house a mile off— I believe he is very reserved & shy & proud or fine—so I suspect he will be no catch, & will never honour us".