The son of sugar magnate John Bundy Brown and Anna Matilda Brown (née Greely), he was born in Portland, Maine on December 14, 1838.

At Bowdoin, he was a student of Joshua Chamberlain, who later was his commander during the Civil War.

In August 1862, while studying for the bar, he was commissioned first lieutenant and adjutant of the 20th Maine Infantry Regiment.

His regiment fought at the battles of Antietam, Fredericksburg, Chancellorsville, and Gettysburg among others.

He was promoted to Lieutenant Colonel of the 32nd Maine Infantry Regiment before being injured and discharged in 1864.