[6] While at Harvard, Brown earned money as the college organist and unhappily spent some summer months as a village school teacher.

[4] To restore his health after years dedicated to study, Brown spent the summer of 1852 working aboard a fishing boat, sailing out of Gloucester, Massachusetts to Prince Edward Island.

In his Autobiographical Notes, Brown wrote that a college graduate in his circumstances had three career choices: the ministry, medicine or the law.

[17] In that year he began developing a small portfolio of clients of his own and supplemented his income with work as organist and choir director in the Episcopal Church in Newton, Long Island.

He remained in that partnership (later named Stanley, Brown & Clarke when Langdell left to become dean of Harvard Law School) until his judicial appointment in 1881.

[19][1] Brown stated that when deciding on a path after college, he found a business career not to his taste, inasmuch as he had no interest in "mere wealth" or a "life of money-making.

In the late 1850s he began investing in and doing legal work for real estate transactions in which large areas of land at the edges of development in New York City were subdivided and sold at considerable profit.

[4][25] His most famous case involved the libel charges against journalist Charles Anderson Dana brought by the administration of Ulysses S. Grant.

[27] Obituaries noted his versatility[28] and compared him to polymath poet and doctor Oliver Wendell Holmes Sr.[27] Modern sources have also recognized his wide range of pursuits and accomplishments.

[29]In 1875, Brown joined the Torrey Botanical Club of Columbia College in New York and was an active member for many years, serving as president from 1893 to 1905.

[31] He wrote that organization's charter in 1891[1] and in that year donated the initial $25,000[30] (which he viewed as "quite out of proportion to my means at that time")[31] toward the $250,000 in private seed money required pursuant to the New York legislature's authorization for municipal contributions.

[37] Brown's Colorado mountaintop observations of the July 29, 1878 solar eclipse were included in a report of the United States Naval Observatory.

[38] Brown met his first wife, Mary Chadwick Barrett, in 1846 at Bradford Academy, near West Newbury, as he studied to prepare for college.

"[42] Others described surprise that Brown, active in numerous elite New York social clubs, managed to keep the impending wedding secret.