Sherborne was the son of James Dutton, 1st Baron Sherborne, of Sherborne, Gloucestershire, by his wife Elizabeth Coke (1753–1824), daughter of Wenman Coke and Elizabeth Chamberlayne.
[2] As a result of the Slave Compensation Act 1837, Sherborne was given £3,579 in compensation from the British government as the executor of Sir Rose Price, 1st Baronet, his brother-in-law; Price's estate included the "Worthy Park" slave plantation in Saint John Parish, Jamaica.
[3] Lord Sherborne married Mary Legge (1780–1864), daughter of Henry Bilson-Legge, 2nd Baron Stawell and Mary Curzon, on 11 August 1803.
They had six children: Lord Sherborne died on 18 October 1862, aged 83, and was succeeded in the barony by his son, James.
Lady Sherborne died in October 1864.