At age 19, he was hired as a studio singer with CBC in Toronto and in 1948, he started a three-year affiliation with Tommy Dorsey, initially as a member of the vocal group Bob-O-Links.
Duffy was performing as a member of the musical act the Town Criers in 1950 and would frequently appear on CBC-TV variety shows through the 1950s.
In 1957, he was hired by Norman Jewison to appear as a comedian on the CBC series Showtime.
Duffy battled alcoholism after he started drinking while on the road with the Dorsey band.
In 1971, he also hosted the CBC-TV series In The Mood, featuring appearances from some of the biggest names in big band jazz, including Benny Goodman and Count Basie.