Beginning as a merchant trader, first of linen in Belfast, then of cotton and tobacco after migrating to Baltimore, Maryland, he later shifted his focus to financial services.

While there is no evidence that Brown was involved in the Irish Rebellion of 1798, he did go into hiding in its aftermath and, perhaps as a consequence of the resulting political and economic conditions in Ireland, he emigrated to the United States.

This was the beginning of a thriving business for Alexander Brown & Company, importing general merchandise and Irish linens to Baltimore, the latter supplied by his cousin William Gihon and friends in Ulster, and exporting tobacco and cotton to Liverpool from the American south.

The Liverpool firm first functioned as an agent for Alexander Brown & Sons, purchasing linen and other goods and becoming a leading seller of cotton and tobacco.

As Brown began to provide financial services for other merchants, buying sterling bills from cotton and tobacco exporters in the American south and selling them to importers of British merchandise in Baltimore, the Liverpool operation became a vital link in the process of handling currency exchange and of providing credit for international commerce, that is, accepting (guaranteeing) bills of exchange, issuing letters of credit, and furnishing information on the financial reliability of firms engaged in the transatlantic trade.

He appointed agents in the key southern cities of Petersburg, Charleston, Savannah, Huntsville, Mobile and New Orleans to facilitate the export of cotton.

Baltimore had flourished during the years of the Napoleonic Wars; however, better port facilities and better access to inland markets made Philadelphia and New York increasingly thriving business centres for both imports and exports during the 1820s and 1830s.

He retired from the bank in 1839 to pursue private financial affairs and to support such philanthropic activities in Baltimore as the Peabody Institute; he died on 26 August 1859.

He remained active in the affairs of Brown Brothers & Company till his death on 1 November 1877, during which time the New York bank emerged as the headquarters of the several firms.

James Brown was a trustee of the New York Life Insurance Company and the Bank for Savings; he took a particular interest in Union Theological Seminary, of which he was a director, and he was involved in numerous charities.