Educated at Eton and Christ Church, Oxford where he matriculated in 1761,[3] he was also a reader of poetry, his favourite poet being John Milton.
Lyttelton received his pension through his estranged father, and because of his parentage and ability, he also had a career in public life.
He was a Whig MP for Bewdley from 1768 to 1769 and the Chief Justice of the Eyre in 1775, and became a Privy Councillor the same year.
Lyttelton married Aphia Witts, but they had no children, so on his death, his barony became extinct.
It was recreated in 1794, however, for his successor in the baronetcy, his father's brother Sir William Lyttelton, 7th Baronet.