He was born in London, the son of François-Marie-Joseph-Justin, comte de Viry [fr] by his first wife, the Englishwoman Henrietta Jane Speed.
[1] In 1776 he joined the chevau-légers of Savoy[1][2] and in 1789 went to England to serve as equerry to the Duke of Gloucester, a post he would hold until 1792.
In July 1795 he fled while on trial for fraud in the Court of King's Bench, and in December, having taken refuge in the Isle of Man, he pleaded parliamentary privilege after being arrested for debt by a Liverpool merchant.
[4] Speed's father had supported Napoleon Bonaparte, and so had retained his estates at Viry, which passed to his son in 1813.
[1] The comte de Viry returned to France on the Bourbon restoration, and on 22 August 1815 was elected deputy for Mont-Blanc.