He was the younger brother of Jimmy Ruffell, who made more than 500 league appearances for West Ham United and won six caps for the England national football team.
However, over the following two seasons he was unable to break into the first team and he subsequently joined Football League Third Division North side Nelson in August 1927.
He made his debut for Nelson in the 2–1 away win against Southport on 17 September 1927, taking the place of Henry White who was dropped despite starting the first four matches of the season.
Ruffell retained his starting berth for the following game, although he was moved to inside-left, as Nelson recorded a 2–1 home victory over Durham City.
He played his twelfth and final league match for the club on 14 January 1928 as the team suffered another heavy defeat, losing 1–5 to Halifax Town.