Having lived in several places in the vicinity of Glasgow during his childhood, Gray began his career at Govan Columbia juveniles and Ashfield in the junior leagues, before joining Edinburgh side Hibernian in 1899.
[5] In the summer of 1904 Gray moved south to London to join Woolwich Arsenal, who had just been promoted to the First Division for the first time.
Gray made his debut in Arsenal's very first First Division match, against Newcastle United on 3 September 1904 and immediately became the club's first-choice right back for the next five seasons; Woolwich Arsenal reached the semi-finals of the FA Cup twice (in 1905–06 and 1906–07), while in the League they usually occupied a comfortable spot in mid-table.
Gray lost his regular place to fellow Scot Duncan McDonald in 1909, but he returned to the team during the 1910–11 season.
However, having passed the age of 30 he found his place came under threat from younger players such as John Peart, and he was largely left out in the 1911–12 season.