His paternal grandparents were Eleanor (née Van de Weyer) Brett and Reginald Brett, 2nd Viscount Esher, an MP and the Constable and Governor of Windsor Castle and a close friend and adviser of Edward VII and George V.[4] His maternal grandparents were Anna (née Atkins) Heckscher and August Heckscher (1848–1941), a German-born American capitalist and philanthropist.

[3] He proceeded to the Architectural Association, but left to learn from the traditionalist A. S. G. Butler and then, as a non-qualified partner of William and Aileen Tatton Brown, passed the RIBA external exams in the summer of 1939, winning the Ashpitel Prize.

[6] In November 1957, some 50 of Hatfield's two-storey terraced houses lost their mono-pitched roofs in a storm, and the adverse publicity and financial liability ended his business.

[3][8] National Life Stories conducted an oral history interview (C467/14) with Lionel Gordon Baliol Brett Esher in 1997 for its Architects Lives' collection.

[12] Esher's autobiography Ourselves Unknown records how he nursed his wife through a long mental breakdown in the 1960s, but notes that she gave him equal support and strength over nearly 70 years.