After serving three years in the US Navy, Duff studied at Stephen F. Austin State College in Nacogdoches and played basketball.
Duff was heard in shows like the Louisiana Hayride, Red Foley's Ozark Jubilee, Saturday Night Shindig and the Grand Ole Opry.
The title of his hit changed over time to "Y'All Come" and was chosen as the theme song by the Houston Hometown Jamboree.
Other artists including Bing Crosby, Patti Page, Bobby Bare, Faron Young, Grandpa Jones, George Jones, Bobby Vinton, Minnie Pearl, Glen Campbell, Buck Owens, Porter Wagoner, Gene Pitney and Little Jimmy Dickens recorded their version of the song.
While working at a radio station in Colorado Springs, he made some minor records for the small Smartt label.