As commander of Special Detachment (Sonderkommando, also known as Einsatzkommando or EK) 1b, he was responsible for mass murder in the Baltic states and Belarus.

Ehrlinger was the son of the mayor of Giengen an der Brenz, a small town in southwestern Germany, in what is now the state of Baden-Württemberg.

The nationalist and xenophobic atmosphere at the University of Tübingen (already by 1931 there were no longer any Jewish professors there) fit in well with his later legal career in the SD, the RSHA, and the Einsatzgruppen.

According to his SA certificate of good conduct, "Ehrlinger was one of the few Tübingen-connected students, who even before the seizure of power put himself regularly where he was needed with the propaganda or other service."

After the beginning of the German invasion of the Soviet Union, on 22 June 1941, Ehrlinger's unit, 70 to 80 men strong, followed behind Army Group North in the Baltic states and the area south of Leningrad.

In September 1943 Ehrlinger was promoted to SS-Standartenführer (colonel) and went to Minsk where he succeeded Curt von Gottberg as SS and Police Leader for Generalbezirk Weißruthenien.

On 1 April 1944, he became chief of Department I (Personnel) in the Reich Security Main Office (Reichssicherheitshauptamt or RSHA), replacing Bruno Streckenbach.

Ehrlinger decided to reduce the alimony payments, making his wife concerned that he was about to emigrate and prompting her to denounce him to the police.