Berger was born in Cleveland, Ohio, to a Hungarian-born railroad engineer and a third-generation German-American Jewish mother from Texas.
During his childhood, his family attended the Euclid Avenue Temple (Anshe Chesed Congregation), where Rabbi Louis Wolsey encouraged him to pursue a career in the rabbinate.
Berger became a prominent figure in the Council’s opposition to the Biltmore Conference of 1942, which formalized Zionist goals for a Jewish state in Palestine.
[4] Berger and the ACJ faced growing opposition after The Holocaust, the establishment of the State of Israel in 1948, and an increasing popularity of Zionism among American Jews and Jewish institutions.
[5] Murray Polner wrote that the majority of American Jews viewed the ACJ as indifferent or hostile to Holocaust survivors who moved to Israel.
In a July 1967 interview, Berger named six prominent Jews—Donald S. Klopfer, Stanley Marcus, John Mosler (chairman of the Mosler Safe Company), Walter N. Rothschild Jr. (president of Abraham & Strauss), Stanley Marcus, Joseph H. Louchheim (deputy commissioner for the New York State Department of Welfare’s New York City division), and Henry Loeb of Loeb, Rhoades & Co.—as supporting his position.
[10] In 1968, upon resigning as the ACJ's executive director, Norton Mezvinsky claimed that Berger had assisted Arab envoys at the United Nations with speechwriting, specifically naming George Tomeh of Syria.
Mezvinsky wrote a detailed obituary that concluded:[12] In 2011, Jack Ross published a biography titled Rabbi Outcast: Elmer Berger and American Jewish Anti-Zionism.
In a statement released to the press, he called for the dissolution of the Council and pleaded for an effort to heal all wounds in order to strengthen Israel by creating a united spiritual front of American Jews.