He graduated from Amherst College in 1855 and moved to Maine, where he was the principal of the Washington Academy in East Machias.

Early in the Peninsula Campaign (April 28, 1862), he was promoted to brigadier general and assumed command of the 1st Brigade of the 1st Division of II Corps of the Army of the Potomac after the brigade commander, Brigadier General Oliver O. Howard, was wounded at the Battle of Seven Pines.

At the Battle of Glendale, he displayed "personal gallantry" in coming to the aid of the beleaguered Union division led by Philip Kearny.

At the Battle of Antietam, on September 17, 1862, he briefly led his division when its commander, Major General Israel B. Richardson, was wounded.

At the Battle of Chancellorsville the following May, his brigade performed well under difficult circumstances, covering the retreat of the army from the crossroads at the Chancellor House.

Caldwell received criticism from Maj. Gen. George Sykes, the commander of the Union V Corps, who was operating nearby.