He was the eldest child of the 2nd Earl and his wife Arabella Mainwaring-Ellerker (died 1782).
On 21 July 1818 he married Mary Fludyer, eldest daughter of George Fludyer of Ayston, County Rutland, esquire and of Lady Margaret Fane, daughter of the 9th Earl of Westmoreland.
He had a large Napoleonic collection and reportedly, on visiting the Louvre with Paul Delaroche in 1848, he commented on the implausibility and theatricality of David's painting Napoleon Crossing the Alps, which had recently been reinstalled.
[1] He commissioned Delaroche to produce a more accurate version which featured Napoleon on a mule, entitled Bonaparte Crossing the Alps.
Stephen Bann suggests that Arthur George's meeting with Delaroche may have occurred, but Delaroche chose to produce two works that are almost identical and send one to America.