[1] Grafton was born at Euston Hall near Thetford, the eldest son of the Reverend Lord Charles Edward FitzRoy and of his wife, Hon.

In 1914, soon after the outbreak of the Great War, he went to France, and in 1917, he was appointed as aide-de-camp and comptroller to Lord Buxton, Governor General of South Africa.

From 1927 to 1936, he was land agent for his first wife's maternal uncle, Owen Hugh Smith, at Langham in Rutland.

In 1936, he succeeded his first cousin John FitzRoy as Duke of Grafton and inherited the family estates based at Euston Hall.

They had three children:[2] Lady Doreen died of septicaemia weeks after the birth of their third child.