He attended schools in Windham and Windsor Counties and at age 15 he became a clerk in a Boston dry goods commission warehouse.

[1] In the mid-1830s Baxter returned to Vermont, after securing the financing to open his own dry goods store in Bellows Falls.

Drawing on these traits and his experience managing employees in Boston, in the early 1840s Baxter bid successfully for the contract to grade rail bed and lay track for the Rutland and Burlington Railroad.

[7] He took steps to prepare the militia in anticipation of the Civil War, and also served as a Delegate to the Peace Conference that attempted to prevent the conflict.

[8][9][10] After the war started Baxter was responsible for recruiting the 1st Vermont Infantry, a regiment enlisted for three months' service.

[11][12] In 1861 he relinquished the Adjutant General's position in favor of someone with combat experience, and was succeeded by Peter T. Washburn, who had been the lieutenant colonel of the 1st Vermont and its de facto commander.

An associate of Cornelius Vanderbilt, he was a major investor in the New York Central Railroad, serving as president from 1867 to 1869, and a member of the board of directors from 1869 until his death.

Baxter's other holdings included large ownership stakes in and/or board of directors memberships with: the Chicago & North Western Railroad; Emma Silver Mine; Pacific Mail Steamship Company; Hannibal & St. Joseph Railroad; Panama Canal Railway; Continental Bank of New York City; and the Pullman Palace Car Company.

[18] Baxter was in increasingly poor health in his later years, especially after an 1877 fall left him with a debilitating back injury.