[4] Born in Raymond, Mississippi,[4] he was the older brother of the blues accompanist Papa Charlie McCoy.

As a young man, McCoy was drawn to the music scene in Memphis, Tennessee, where he played guitar and sang during the 1920s.

[4] After they were divorced, McCoy teamed up with his brother to form the Harlem Hamfats, a band that performed and recorded during the second half of the 1930s.

McCoy's songs have also been covered by Skip James, Bob Dylan, John Mellencamp, the Ink Spots, Ella Fitzgerald, Jo Ann Kelly, Cleo Laine and A Perfect Circle.

A tribute concert[9] was held in October 2010 to celebrate the music of Joe and Charlie McCoy and to buy gravestones for each of them; they were installed on May 31, 2011.