The son of John Egerton, 4th Earl of Ellesmere, and Lady Violet Lambton, he was educated at Eton and Trinity College, Cambridge.

In 1963, George Sutherland-Leveson-Gower, 5th Duke of Sutherland, his distant cousin "Geordie", died, leaving no immediate male heir.

The wealth had come from the acquisitions of the third Duke of Bridgewater, who built the famous canal and passed on his mining riches, and from intermarriage.

The Scottish National Gallery in Edinburgh had a number of notable paintings in its possession on long-term loan from the Duke of Sutherland's estate, including pieces by Titian, El Greco, Raphael and van Dyck (one of which, the Venus Anadyomene, was bought by the gallery after his death towards inheritance tax).

Following the death of his first wife, on 16 August 1979 he married Evelyn Moubray, a former interior decorator who shared his love of fishing.