During his time at university he participated in Varsity Matches in five sports: cricket (1876–79), football (1876–78), athletics (1876; selected to throw the hammer), rackets (1877–79) and real tennis (1877–79), displaying an ability that made him arguably the pre-eminent sportsman of his generation; his only rival in terms of versatility was Oxford's Cuthbert Ottaway.

In the course of this game, Australia reached a total – all but unheard-of at the time – of 500 for six and the England captain, Lord Harris, grew so desperate for a breakthrough that he asked his wicket-keeper, Lyttelton, to bowl.

Lyttelton was also a keen and skilful footballer, playing for Cambridge and Old Etonians and winning a single cap for England against Scotland on 3 March 1877.

Once playing against Royal Engineers I saw him make a run down from one end of the field to the other and floor four men on the way – the last two having charged him simultaneously from both sides, and both rebounding on their backs – and shoot the goal at the end.Lyttelton's principal weapon as a forward was a unique and generally successful goalscoring technique that appears, from contemporary sources, to have been an early version of the 1970s Cruyff turn.

This continued till he got opposite the further goal post, and then one huge foot was smartly dropped on the ball, stopping it dead, and of course the pursuers all ran a yard or so too far, not suspecting the sudden pull up; thus he had a clear shot at the goal.Lyttelton played in the last days of the "dribbling game", the earliest form of the Association code and a style of play that valued individualism and close ball control over passing and teamwork.

In his solitary international his England teammates were highly critical of his attempts to dribble through the entire Scotland defence by himself, and Billy Mosforth, the Sheffield player, ventured to draw his colleague's attention to this failing.

He remained apolitical until his uncle's retirement, but in 1894 entered politics as a Liberal Unionist, and was elected to the House of Commons at an 1895 by-election as Member of Parliament for Warwick and Leamington.

His decision to allow Chinese indentured labour into South Africa proved highly controversial and became a major issue for the Liberals in the 1906 election.

Lyttelton also attempted ambitious reforms of Britain's management of the colonies towards a more decentralised, imperial vision, but these ideas were abandoned by the Liberals who succeeded him.

His funeral took place on the day of the annual Varsity cricket match, and play between Oxford and Cambridge was suspended for two minutes as a mark of respect.