Brown was serving an apprenticeship in the linen trade and playing football for his works team, William Ewart & Son F.C., when he was spotted by Belfast Celtic during the early 1930s.
In December 1934, shortly after his 20th birthday, Brown signed for English First Division side Wolverhampton Wanderers, making his debut on 12 January 1935 in a 4–0 win over Notts County in the FA Cup.
He made his last appearance for the club against Bristol City on 5 May 1951 When Brown began his international career in 1935 there were, in effect, two Ireland teams, chosen by two rival associations.
Brown played regularly for the IFA XI before the outbreak of the Second World War, making his last appearance on 15 March 1939 in a 3–1 defeat to Wales.
In 1937, while playing for Coventry City, Brown also made 2 appearances and scored 1 goal for the FAI XI (forerunners of the Republic of Ireland national football team).
However this time the IFA objected and all three players received telegrams from the English FA ordering them not to accept the offer on the grounds they were not born in the Irish Free State.