[2] Though it has been claimed that his early life was obscure and that he suffered "fatal neglect" in formal education,[3] it appears that he was educated in Inverness and that he remembered Simon Fraser, 11th Lord Lovat, visiting his mother to seek the support of the Baillies for the Jacobite cause and later witnessing the battle of Culloden from the hill above Dochfour.
[2] He was first in the West Indies in 1759–60, serving with the 4th Foot in Martinique, as his 1835 Inverness Courier obituary makes clear.
Baillie himself seems to have been a staff officer to General William Howe at the Siege of Havana in the adjutant's department.
One of their business connections was with Henry Laurens of Charleston in South Carolina with whom they exchanged African slaves and plantation produce for necessary stores.
[3] During the French Revolutionary War Baillie raised the Bristol Volunteer Infantry in 1797 and served as their Lieutenant-Colonel (Colonel in 1798).
He noted how feeble were attempts in parliament to oppose slave trade abolition bill but he was not among those who made last stand against it.
By 1811, he was suffering poor health and was concerned for his son Peter (by then MP for the Inverness burghs) who died in 1812.