From the mid-eighteenth century, it was the residence of military topographer and artist Thomas Sandby (the brother of the better-known Paul), serving as Deputy Ranger of the Great Park.

It was expanded in 1792 and became the residence of Joseph Frost, the Park Bailiff, and later of the General Superintendent of Farms following Sandby's death.

The grounds extend to 98 acres (40 hectares),[3] partly under its own head gardener, but primarily the responsibility of the Crown Estates Commissioners.

While the house has been expanding gradually since the 1840s, it remains relatively small and informal, yet the grounds follow a cohesive plan.

This was the result of work undertaken by the Duke and Duchess of York in the 1930s, with the assistance of Sir Eric Savill, of the Windsor Estate.

The grounds contain the miniature cottage Y Bwthyn Bach, a gift to Princess Elizabeth as a child from the people of Wales in 1932.

[6] In August 2003, Prince Andrew, Duke of York was granted a lease agreement by the Crown Estate for 75 years.

The lease agreement required Prince Andrew to undertake refurbishments at his own expense, which was underestimated at £7.5 million at September 2002 prices, excluding VAT.

[3] The lease agreement provides that the prince may not benefit financially from any increase in the value of the property, as the freehold belongs to the Crown Estate.

In the circumstances, the Crown Estate considered that the requirement to obtain value for money was satisfied, taking into account the non-financial considerations relating to the lease of the property.

[9] In 2023, reports suggested that King Charles III was to cut Andrew's annual grant, potentially leaving him unable to afford the Lodge's running costs, and he had been offered the smaller five-bedroom Frogmore Cottage instead, until then the UK residence of Prince Harry, Duke of Sussex and his family, who had been issued a request to vacate by Buckingham Palace.