His mother was a sister of John M. Mason, D. D. Brown was reared at Liberty Hall, his parents' estate in Frankfort, Kentucky.
After a few years he removed to Frankfort and became the law partner of Benjamin Mills, and subsequently, of Charles S. Morehead.
[1] Following his first wife's death, Brown married Mary Yoder of Spencer County, Kentucky, in 1835.
[1] In 1839, Brown was appointed to the bench of the Circuit Court of his District, and he served in that capacity until resigning in 1849 to return to his law practice.
[1] Morehead was elected governor of Kentucky in 1855 and appointed Brown as his Secretary of State.