His father, who was the Sheriff of Derbyshire in 1730, was the son of Lieutenant Colonel Robert Mundy and his wife Ellen (née Slack).
He was a direct male-line descendant of Sir John Mundy, who had purchased the manors of Markeaton (the principal seat of the Mundy family), Allestree and Mackworth from Lord Audley in 1515, he was later appointed Lord Mayor of London in 1522.
Before her death on 28 October 1783, they were the parents of five sons and one daughter:[1] On 14 January 1788, he married Georgiana, Lady Middleton as his second wife.
Before her death on 29 June 1789, they were the parents of one daughter:[1] On 19 October 1811, he married, as his third wife, Catherine (née Coffin) Barwell, the widow of Richard Barwell, MP for Helston and St Ives, and daughter of Nathaniel Coffin of Bristol.
Through his daughter Frances, he was a grandfather of Sir Charles Augustus FitzRoy, who served as Governor of New South Wales, Prince Edward Island and Antigua; he married Lady Mary Lennox, the eldest child of Charles Lennox, 4th Duke of Richmond.