Frustrated with the racial prejudice he experienced in the United States as a black man, Spencer moved to West Germany in 1950, where he had a successful singing career.

He took private vocal lessons while working as a gardener, and eventually caught the attention of the tenor Roland Hayes who helped him to get a scholarship at the Eastman School of Music.

In 1938 he sang in the Federal Music Project NBC Blue radio opera Gettysburg, first from El Capitan Theatre in Los Angeles, then at the Hollywood Bowl.

Spencer was so enamored with the German public and frustrated with the meager opportunities he found as a black artist in America that he moved his family to Wuppertal, West Germany in late 1950.

His ability to perform not only Spirituals and classical music, but also folk songs in their original languages (French, German, Italian, Russian, Hebrew) won him much popularity in France and post-war Germany.