Early in 1970, Coe released his blues-oriented debut album, Penitentiary Blues, followed by a tour with Grand Funk Railroad.

The Mysterious Rhinestone Cowboy was produced by Rod Bledsoe and boasted a coterie of Nashville’s top session musicians, including steel player Pete Drake, pianist Hargus “Pig” Robbins, and multi-instrumentalist Charlie McCoy.

[4] and observes: Opening with "A Sad Country Song," Coe displays his lyrical and melodic gift that comes out of the great Texas and Bakersfield traditions.

Similarly, in the prideful “I Still Sing the Old Songs,” the grandson of a Confederate soldier pays tribute to his grandfather with the lines: The narrator also speaks of his father, who “wore his purple heart so proudly,” and his own “turn behind the gun,” presumably in Vietnam.

The Mysterious Rhinestone Cowboy contains several cover songs by other like-minded songwriters who were beginning to push the barriers of conventional country music, such as Guy Clark (“Desperados Waiting for a Train”) and Mickey Newbury (“33rd of August”).

Newbury in particular was a major influence on Coe, with Coe dedicating The Mysterious Rhinestone Cowboy to the Texas songwriter and including a letter to Newbury on the back cover, part of which observes, “Freedom is knowing how to remember the weight of your chains once they’ve been removed, for each man feels his own pain in prison and each man must pull his own time.” Allmusic reviewer Thom Jurek describes the album as being "as good as anybody at his best", writing "Coe displays his lyrical and melodic gift that comes out of the great Texas and Bakersfield traditions.

"[4] Rolling Stone ranked it #40 on its “50 Country Albums Every Rock Fan Should Own” list, stating, “This major-label debut, which reflected a time when Coe wore rhinestone duds, wigs and a mask onstage while living in his car (a white hearse), posits him as a doomed, lonely troubadour of the lost-cause South (‘I Still Sing the Old Songs,’ ‘Old Man Tell Me,’ ‘The Old Grey Goose is Dead’).