In 1819, after losing interest in a mercantile career in Troy, New York, he entered the United States Army as a second lieutenant in the 2nd US Infantry Regiment on March 3, 1819.

Sumner's military appointment was facilitated by Samuel Appleton Storrow, Judge Advocate Major on the staff of General Jacob Jennings Brown of the Northern department.

They had six children together: Nancy, Margaret Foster, Sarah Montgomery, Mary Heron, Edwin Vose Jr., and Samuel Storrow Sumner.

Lt. Gen. Winfield Scott assigned Sumner as the senior officer to accompany Lincoln from Springfield, Illinois, to Washington, D.C., in March 1861.

[8] When Sumner left for California, his son-in-law Armistead Lindsay Long resigned his commission and enlisted with the Confederate Army eventually becoming Robert E. Lee's military secretary and an artillery brigadier general.

The II Corps, commanded during the war by Sumner, Darius N. Couch, Winfield Scott Hancock, and Andrew A. Humphreys, had the deserved reputation of being one of the best in the Eastern Theater.

"[10] At the Battle of Seven Pines, however, Sumner's initiative in sending reinforcing troops across the dangerously rain-swollen Chickahominy River prevented a Union disaster.

Despite his old-fashioned ideas on discipline and respect for commanding officers, the II Corps troops generally had a positive opinion of him.

The assault was devastated by a Confederate counterattack, and Sedgwick's men retreated in great disorder to their starting point with over 2,200 casualties.

Historian M. V. Armstrong's recent scholarship, however, has determined that Sumner did perform appropriate reconnaissance and his decision to attack where he did was justified by the information available to him.

Maj. Gen. William B. Franklin wanted to attack with his fresh VI Corps, but Sumner, who was senior to him, ordered him to hold back.

In this capacity, he took part in the disastrous Battle of Fredericksburg, in which the II Corps, now commanded by Major General Darius N. Couch, suffered heavy casualties in frontal assaults against Confederate troops fortified at Marye's Heights.

The Onondaga County Civil War Round Table was raising funds to repair the grave and the general area.