Competing in the 200 m breaststroke, he won the European title in 1950 and an Olympic bronze medal in 1952.

In the Olympic final, Klein was third behind Australian John Davies the gold medalist and American Bowen Stassforth.

His world record in the 200 m breaststroke (long course) of 2:34.4 was equaled by John Davies in the final.

Davies, Stassforth, and Klein were the only three swimmers in history to better 2:35 in the 200 m breaststroke (long course) prior to the bifurcation of the stroke in 1953.

The 1952 US Olympic Book lists the 200 meter breaststroke world record as belonging to Herbert Klein with a time of 2:27.3 who swam it in a short course pool.