Harrowby was a descendant of Thomas Coutts and of Sir Dudley Ryder, Lord Chief Justice of the King's Bench in the 1750s.
He moved to the 133 Field Regiment Royal Artillery in February 1945, part of the 53rd (Welsh) Infantry Division, but was severely injured near the Reichswald forest only a few days later.
He remained in the Far East after the war, serving as a political officer with the 5th Battalion of the Parachute Regiment in Java, under Laurens van der Post.
[citation needed] Although he did not follow many of his ancestors in standing for Parliament, Harrowby was a councillor in the Royal Borough of Kensington and Chelsea from 1950 to 1971.
He was president of the Wolverhampton South West Conservative Association, resigning in protest after the sitting MP, Enoch Powell, made his "rivers of blood" speech in 1968.
He was also a director of the National Provident Institution until 1986 and of the Saudi International Bank, Bentley Engineering, Powell Duffryn, Dowty and Dinorwic Slate Quarries.