Kelly was born in Lurgan and played Gaelic football with Clan na Gael while working in the family drapery firm.
In January 1941, Glenavon suffered a nightmare defeat at the hands of Belfast Celtic, losing 13–0 with Peter O'Connor scoring eleven – this remains an individual British Isles record.
[2] Despite this embarrassing defeat, Belfast Celtic's manager, Elisha Scott (a former goalkeeper), signed Kelly in the summer of 1941 to replace Tommy Breen.
Never afraid to stand-up to authority, when he was reprimanded by Scott with the words, "My grandmother could have played better out there", Kelly adjusted the following week's team-sheet, putting a line through his own name and replacing it with "Lish's granny".
[9] By the end of the season, Kelly had fallen out of favour with the Southampton manager, Sid Cann, following disciplinary problems after a match at Leicester in April 1951.
[3] By now completely out of favour, Kelly was fifth choice 'keeper behind Fred Kiernan, Christie, Len Stansbridge and Eddie Thomas and was unable even to get a match in the reserves,[11] spending the whole of 1951–52 on the sidelines, with some time back in Ireland.