Whereupon, he was commissioned a Second Lieutenant in the Army Artillery Corps and served as an assistant professor at West Point before resigning in October 1820.

His administration had little influence on the course of the Louisiana government, as its orders were liable to be countermanded by the military due to the reconstruction acts.

Baker removed 9 New Orleans City Councilmen which resulted in President Ulysses S. Grant's reversal of this order.

Before her death on August 17, 1831, they were the parents of three children, including: He married a second time to Catherine Patton from Fairfax, Virginia in 1832.

Baker died in Lyme, Connecticut, on April 16, 1885, at "Cricket Lawn" the home of his daughter Margaret Van Bergen.