He was a son of William Duff, 1st Earl Fife, and, his second wife, Jean Grant.
Before his parents' marriage, his father was married to Lady Janet Ogilvy (a daughter of the 4th Earl of Findlater).
[2] Upon the death of his elder brother, James, in 1809, he succeeded to the title of Earl Fife.
[4] Together, they were the parents of two sons and four daughters, including:[5] Lord Fife died on 17 April 1811 at Duff House in Banff.
Through his second son, Alexander, he was a grandfather of James Duff, 5th Earl Fife, who married Lady Agnes Georgiana Elizabeth Hay (a daughter of William Hay, 18th Earl of Erroll and Lady Elizabeth FitzClarence, an illegitimate daughter of King William IV),[10] and George Skene Duff, an MP for Elgin Burghs.