He was a descendant of the poet and adventurer Lord Byron (1788–1824), via his daughter Ada Lovelace (1815–1852), arguably the world's first computer programmer.

In 1925, Lytton and his sister Anne changed their surname to Lytton-Milbanke by deed poll,[3] in honour of Noel's mother's succession to the Wentworth barony, which could pass to either of them.

[4][5] Lytton was raised just east of the Sussex town of Crawley, in the mansion built by his maternal grandparents on the grounds of their renowned horse breeding establishment, the Crabbet Arabian Stud.

[6] When the British entered the Second World War, he was posted by the military to North Africa and Italy, but due to an automobile accident was invalided out to desk duty, which his son describes as extremely frustrating for someone who was used to being athletic and active.

[2][7] As part of government administration, Lytton eventually went to Yugoslavia to work with Josip Broz Tito's Partisans.