Playing as an inside forward, his regular goalscoring for Orient soon attracted the attention of Arsenal, who signed him in November 1955 for £23,000, at the time a very high figure.
He scored on his debut, against Sheffield United on 12 November 1955, in a 2–1 First Division win (Don Roper grabbing the other Arsenal goal).
Despite an eye-catching start, Groves initially struggled to live up to expectations at Arsenal, his first season at the club hampered by injury, first to a knee and then to his back.
The season after that, he was switched to wing-half, where he played for the rest of his career; Groves was also made Arsenal captain, following the departure of Dave Bowen.
Groves lost his first-team place in the 1961–62 season, although he stayed at Arsenal as a bit-part player until the summer of 1964, when he left to sign for Southern League side Canterbury City.