His father was born in 1670 in Ireland, fought for the Jacobites in the Williamite War and had gone to France as the colonel of Dillon's Regiment with the Irish Brigade in April 1690 when Irish troops were sent to France in exchange for French troops sent to Ireland with Lauzun.

His father's family was Old English Irish and descended from Sir Henry Dillon who came to Ireland with Prince John in 1185.

Dillon's Regiment, led by Charles's father, fought for France in the War of the Spanish Succession (1701–1714).

[8] However, in 1713 France signed the Peace of Utrecht, recognising the Hanoverian succession and ending its support for the Jacobites.

[9] James Francis Edward had to leave France and went to the Duchy of Lorraine, then to Avignon, and finally to Italy.

[14] In 1722 James Francis Edward Stuart (the old pretender), who now resided in Rome, created his father Earl of Dillon.

[21][22][c] During the War of the Polish Succession (1733–1735), the comte de Dillon, as he now was, and his younger brother Henry fought under Marshal Berwick, the commander of Louis XV's Rhine army.

[39] As his only son had predeceased him, he was succeeded by his younger brother Henry as the 11th Viscount,[40] and also as colonel-proprietor of Dillon's Regiment.