[1][2] Grafton is the son of James Oliver Charles FitzRoy, Earl of Euston (1947–2009), and his wife, Lady Clare Amabel Margaret Kerr, one of the daughters of the 12th Marquess of Lothian.

[3][2] His ancestor Henry FitzRoy, 1st Duke of Grafton (1663–1690), was a bastard son of King Charles II by his mistress Barbara Villiers.

[4] Grafton shares the surname FitzRoy (meaning "son of the king") with other natural lines descended from Charles II.

[5] Educated at Harrow School and the University of Edinburgh, Grafton spent a post-graduate year at the Royal Agricultural College, Cirencester, studying estate management.

[1][6] From 2002 to 2004 Viscount Ipswich, as the Duke was then known, worked in the United States in music business management, as a radio host in Nashville, Tennessee, and in 2005–2006 as a merchandise coordinator for the Rolling Stones on the A Bigger Bang tour.