Colonel Charles Lennox in contemporary scorecards, Richmond had 55 recorded first-class appearances from 1784 to 1800 and played a few more games after that.

On 27 May 1789, while a colonel in the Duke of York's regiment, he was involved in a duel with Frederick, Duke of York, who had expressed the opinion that "Colonel Lennox had heard words spoken to him at Daughbigny's, to which no gentleman ought to have submitted", effectively an accusation of failing to respond to an insult in the way that a gentleman should.

In 1794 and 1795 he participated in naval engagements against the French in the West Indies and Gibraltar, but was sent home when he came into conflict with his superiors.

On 15 June, the night before the Battle of Quatre Bras, his wife held a ball for his fellow officers.

The glittering celebration became famous as the Duchess of Richmond's ball and was immortalised by William Makepeace Thackeray in Vanity Fair and by Lord Byron in Childe Harold's Pilgrimage.

Although the Duke observed the Battle of Quatre Bras the next day, as well as Waterloo on 18 June, he did not participate in either, his role being in the defence of the city of Brussels.

During the summer of 1819, Richmond was undertaking a tour of Upper and Lower Canada on an inspection of a planned route for the Rideau Canal, when he was bitten on the hand by a fox.

The injury apparently healed and he continued on the tour, but later in his journey, the initial symptoms of hydrophobia appeared, a clear sign of rabies.