He was the eldest son of Lord Henry Crichton-Stuart, third son of John Stuart, 1st Marquess of Bute (see the Marquess of Bute for earlier history of the Stuart family).
His mother was Lady Gertrude Emilia, daughter of George Mason-Villiers, 2nd Earl Grandison (see the Viscount Grandison for earlier history of this family).
There was uncertainty over the validity of his marriage and his son Henry Villiers-Stuart was not allowed to succeed in the title, which became extinct on his death in 1874.
William Villiers-Stuart (1804–1873), second son of Lord Henry Crichton-Stuart and younger brother of the first Baron, sat as Member of Parliament for Waterford.
His grandson William Desmond Villiers-Stuart (1872–1961) was a Brigadier-General in the British Army and fought in the First World War.