At age 18 he left the farm to learn the carpentry trade, and in 1833 moved to Chicago, Illinois to work as a builder.
In 1834, he came to Milwaukee in the Wisconsin Territory at the request of Solomon Juneau, making the trip together with Horace Chase and Morgan L. Burdick.
[2] By 1842, he had settled on an extensive farm in the area including what is now Johnson Park on Fond du Lac Avenue between 17th and 20th Streets, where he would live the rest of his life.
In July of that year, this farm was a predecessor to the Underground Railroad, and provided refuge to Caroline Quarlls, a 16-year-old runaway slave from a plantation in St. Louis, and the first documented of many such fugitives who would escape to freedom in Canada through Wisconsin.
Perhaps no person has ever lived in Milwaukee with so positive character as Deacon Brown, who had so few enemies, or to whom more people have applied in the hour of trouble, domestic or pecuniary, for advice, than to him.