He was the son of the 2nd Earl of Ashburnham and the former Elizabeth Crowley, being styled Viscount St Asaph from birth, and was baptised on 29 January 1761 at St George's, Hanover Square, London, with King George III, the Duke of Newcastle and the Dowager Princess of Wales as his godparents.

In 1780, Lord St Asaph graduated from Trinity College, Cambridge, with a Master of Arts degree.

[2] He married, firstly, Sophia Thynne (19 December 1763 – 9 April 1791), daughter of the 3rd Viscount Weymouth (later the 1st Marquess of Bath), on 28 August 1784.

They had four children: He married, secondly, Lady Charlotte Percy (3 June 1776 – 26 November 1862) on 25 July 1795.

She was a daughter of the 1st Earl of Beverley, and a sister of George Percy, 5th Duke of Northumberland.