He was named after his maternal grandfather, Associate Supreme Court Justice Nathan Clifford, who had also been a U.S. Representative for Maine, an ambassador to Mexico and an attorney general for Democratic President James K.
He collected birds and attempted taxidermy several times before entering Bowdoin College in 1873.
However, poor health interrupted his studies, and it made it difficult for him to find employment.
In 1874, Brown sought Charles Johnson Maynard to study taxidermy, which jump-started his ornithological career.
In 1881, he was named curator of ornithology for the Portland Society of Natural History, a position he held until 1889.