After nine years' service, his wife's health failing, Brown returned to the United States and became a pastor at Sand Beach Church and teacher of boys at Owasco Outlet, near Auburn (1851–59).

[2] On arrival, Brown shared residential accommodation with the family of the Presbyterian medical missionary Dr. James Curtis Hepburn, then residing at Jobutsuji in Kanagawa, a dilapidated temple formerly occupied by the Dutch consulate.

[5] In July 1860, at the request of English-speaking merchants in Yokohama, Brown begun to preach regularly at Sunday morning service that attracted 30 to 40 congregants each week.

In 1861 Brown also contributed to drawing up the plans and specifications for the British Anglican Garrison Church built on Lot 105 in the foreign settlement.

The Niigata sojourn was only brief; desiring to be close to his fellow New Testament translators, Brown accepted a new teaching post and relocated back to Yokohama in 1870.