In 1927, Brown left Scotland to search for his father, settling in Westfield, New Jersey and finding work on the production floor of a metal box factory, where his riveting skills were handy.

Their uncle Alex Lambie (husband of their mother's sister) was an imposing professional center-half and captain of Partick Thistle in the 1920s.

On his return from Uruguay, he rejoined his team that was renamed the New York Soccer Club, where he scored 6 goals in 25 appearances.

He then moved to the Brooklyn Wanderers for the 1931 spring season with his old teammate, Razzo Carroll, where James scored 10 goals in 31 appearances.

Brown moved to the Newark Americans in the fall 1931 season, but by this time the ASL was collapsing, and he played 13 games and scored 7 goals.

While Brown scored regularly with United, he alienated the team management with his outspoken support for a players' union.

He made only one appearance for the first team,[4] but scored 53 goals in 74 games for the reserves and won the 1934–35 London Challenge Cup.

He was loaned out to Brentford's First Division team, to play center-half in the FA Cup match against Fulham, where he scored a goal.

He then moved north to finish his career with Scottish First Division club Clyde, signing in November 1940 and played two matches with his brother John against Queen's Park in February and March 1941.

Following their elimination, the U.S. played a series of exhibition games throughout South America against professional and regional teams in Uruguay and Brazil.

Brown scored one goal in the last exhibition game against Botafogo in Brazil, a 4–3 loss that counted as a full international.

In September 1949, James played one match in an attempted come-back with powerhouse Hispano, alongside a young goalie tending the sticks by the name of Gene Olaff.

[8] Having retired from playing professionally in 1941, Brown resumed his trade as a riveter in the Troon Shipyard and then moved back to the U.S. to coach Varsity soccer and Riflery.