Born in Strasburg, North Dakota, Klein was the son of German-Russian parents and the oldest of five children.
Upon graduating from high school in 1936, Johnny first enrolled at the University of North Dakota.
Later he enrolled at the University of Missouri, where he graduated with both a Bachelor of Arts and master's degree in Music Education.
However, ill health due to a bleeding ulcer forced him to step down full-time in 1976.
He also continued to perform as well, often playing for the Sunday Swing at the Welk Resort in Escondido with a group called the Journeymen and with other clubs until his retirement in the 1990s.