In 1772 he married Charlotte-Sophie de Guérin de La Marche (d-1806); their son, Count Theodor Franz Baillet von Latour, was Austrian minister of war during the Revolutions of 1848, and was murdered during the Vienna Uprising.

His brother, Count Ludwig Wilhelm Anton Baillet de Latour-Merlemont (1753–1836), was also a general in Austrian service, until 1810, when he resigned, and joined the Grande Armée of Napoleon.

In the course of the 1789 revolt that led to the proclamation of the autonomous United States of Belgium, Latour was elevated to the rank of a Feldmarschall-Leutnant.

His troops held the Duchy of Luxembourg and played a significant role in regaining the rebellious provinces by the end of 1790, which earned him the Military Order of Maria Theresa.

[1] In 1805 Emperor Francis I of Habsburg appointed him president of the Hofkriegsrat, Latour nevertheless died shortly afterwards in Vienna.