Nugent was born in Clonyn, County Westmeath, the only surviving son of George Frederick Nugent, 7th Earl of Westmeath, and Maryanne, daughter of Major James St John Jeffereyes and Arabella Fitzgibbon.

His parents divorced in 1796 after his father's discovery of his mother's affair with Augustus Cavendish-Bradshaw, which also resulted in a celebrated action for criminal conversation.

Her unsuccessful efforts to obtain custody of her daughter later led her to campaign for reform of the law together with the prominent writer and feminist Caroline Norton.

[1] He also had an illegitimate daughter, Eliza Nugent (c. 1806 - 14 September 1877), who married Alfred Harley, 6th Earl of Oxford and Mortimer.

Through his other illegitimate daughter (by Maria Joséphine Touret),[4] Cecilia Henrietta Nugent (1849-1886), who married Willem Boissevain (1849-1925),[4] he is an ancestor of Annemarie, Duchess of Parma.