His father had started the business in 1800 importing Irish linen and exporting cotton and tobacco back to Britain and was also instrumental in the Second Bank of the United States.

As his elder brother, a Liberal Member of Parliament, had founded the banking house of Brown, Shipley & Co., George succeeded his father as head of Alex.

[3] In 1818, George partnered with his younger brother John in starting the firm of Brown Bros. & Co. in Philadelphia, Pennsylvania.

[4] These firms were later merged under the name Brown Bros. & Co. and James's son, John Crosby Brown became a driving force for growth, making Wall Street in New York the center for operations and seeing the bank become major lenders to the textile, commodities, and transportation industries.

[3] George Brown had a prominent role in the founding of the Baltimore and Ohio Railroad (B&O), the first common carrier railroad in the U.S.[6] On February 12, 1827, he hosted a meeting at his Baltimore home for 24 leading merchants, where Philip E. Thomas joined Brown in advocating the formation of a railroad to make Baltimore competitive with other eastern seaboard ports.

[7] At a subsequent meeting in Brown's house (near where Baltimore's City Hall now stands), the committee concluded that the then-unproven technology of a long-haul railroad was feasible and that considerable trade "would flow into the State of Maryland, upon the proposed Rail Road," the committee's report concluded.

Together, they were the parents of seven children, four of whom survived to adulthood:[9] Brown died at his residence in Baltimore on August 26, 1859.