First serving with the army on Sardinia from 1783, he returned to France in 1791, enrolling as a private dragoon in the 15e régiment and becoming aide-de-camp to General Dagobert.
Also serving on the French invasion of Egypt, he became aide-de-camp to Napoléon Bonaparte in 1800, général de brigade after the Battle of Marengo, and finally commander of the Légion d'honneur in 1804.
John Relly Beard writes that "It has been suggested that torture [of Toussaint Louverture] was employed by Bonaparte's aid, Caffarelli" although he states that he can find no evidence to support this claim.
There he defeated an attempted English landing at Laredo, seized Bilbao, was named governor of Biscay in 1811, completely routed Mina's band of guerrillas and contributed to the raising of the siege of Burgos.
In 1813 he once again became Napoleon's aide-de-camp, and when the Emperor left his army he entrusted Caffarelli with commanding the remaining troops in Paris, the government of the imperial palace and organising the guard for empress Marie Louise.