Apart from his political career he was also a Captain in the Rifle Brigade and a Trustee of the National Gallery from 1909 to 1925.

The couple had two sons, who were both killed in military service, and three daughters: On 3 June 1919, Lord Ribblesdale married secondly Ava Lowle Willing, daughter of Edward Shippen Willing and Alice Caroline Barton and former wife of John Jacob Astor IV, at St Mary's, Bryanston Square in London.

He died on 21 October 1925, aged 70 at his townhouse in Grosvenor Square, Mayfair, London, and was buried in the Lister vault at St Mary the Virgin Churchyard, Gisburn, Lancashire.

[1][7] Lady Ribblesdale died on 9 June 1958 in her apartment at 720 Park Avenue in New York City and she was buried in Trinity Church Cemetery.

[8] Among his grandchildren were Simon Fraser, 15th Lord Lovat and 4th Baron Lovat (1911–1995), Sir Hugh Fraser (1918–1984), the Secretary of State for Air from 1962 to 1964, David Fane, 15th Earl of Westmorland (1924–1993), Julian Fane (1927–2009), the author.