The house was built as a hunting lodge for George II, by the architect Roger Morris, and construction began shortly after his accession to the throne in 1727.

Old Lodge itself had been built by George II[3] for Britain's first prime minister, Sir Robert Walpole, who frequented it, particularly to hunt at the estate.

[7][8] The prime minister, John Stuart, 3rd Earl of Bute, became ranger of Richmond Park after Princess Amelia's resignation in 1760.

[7] After Viscount Sidmouth died in 1844, Queen Victoria gave the house to her aunt – the last surviving daughter of George III – Princess Mary, Duchess of Gloucester and Edinburgh.

Although the Prince of Wales favoured stimulating company to hard study, Prince Albert kept him here in seclusion, with only five companions, two of whom were tutors, the Reverend Charles Feral Tarver, his Latin tutor and chaplain and Frederick Waymouth Gibbs.

[14] The next occupants of the Lodge were Prince Francis, Duke of Teck and his wife, the former Princess Mary Adelaide of Cambridge, who were given use of the house by the mourning Queen Victoria in 1869.

[14] Princess Mary Adelaide, a granddaughter of George III and therefore first cousin to the Queen, was famous for her extravagance.

[14] In 1894, the Duchess of York gave birth to her first child, the future Edward VIII, at White Lodge.

[14] After Queen Victoria's death in 1901, the Lodge was occupied by Eliza Emma Hartmann, a wealthy widow prominent in London society,[16] who was declared bankrupt in 1909.