Isaac N. Brown was born in Caldwell County, Kentucky, but spent part of his later youth in western Tennessee.

In March 1834, shortly after his father died, he joined the United States Navy as a midshipman.

In May 1862 he was ordered to Yazoo City, Mississippi, to take command of the unfinished ironclad CSS Arkansas and complete her construction in the worst of conditions.

After successfully accomplishing this difficult task, Brown commanded her dramatic breaking of the Union blockade of Vicksburg, Mississippi, on July 15, 1862.

Brown was promoted to commander in August 1862 in recognition of his bold and audacious actions at Vicksburg.