He was disciplined and studious,[1] graduating from the United States Military Academy at West Point, New York on July 24, 1818,[2] as a second lieutenant of light artillery.

[2] Brown also took part in the Second Seminole War as a lieutenant colonel, participating in the Battle of Wahoo Swamp, and was brevetted a major for "gallant conduct" on November 21, 1836.

[5] In January 1861, Brown was appointed military commander of Washington, D.C., and Fort McHenry and remained in this capacity until war broke out four months later.

Placed in command of the expedition to hold Fort Pickens in Confederate held Florida, he successfully defended Santa Rosa Island on October 9[4] and repulsed an attempt by the enemy to capture the fortress during November 22–23, 1861.

[2][6] On April 25, 1862, President Abraham Lincoln nominated Brown for appointment to the grade of brevet brigadier general in the regular army, to rank from November 23, 1861, for "gallantry and good conduct during the engagement of November 22 and 23, 1861 between Fort Pickens and the rebel batteries", and the United States Senate confirmed the appointment on March May 12, 1862.