In 1932, Colmer was elected to the House of Representatives as a Democrat from Mississippi's 6th District, located on the Gulf Coast.

His district was renumbered the 5th after the 1960 Census, when Mississippi's declining proportion of the US population due to the Great Migration cost it a congressional seat.

Originally elected as a supporter of President Franklin D. Roosevelt's New Deal, Colmer became increasingly conservative as the years passed.

After the Brown v. Board of Education (1954) decision by the United States Supreme Court, ruling that public school segregation was unconstitutional, Colmer helped to get Southern Democratic congressmen to sign the "Southern Manifesto"[1] declaring their resistance.

[7] Colmer endorsed the unpledged electors slate in 1960, Republican Party presidential candidates Barry Goldwater in 1964, and Richard Nixon in 1972.