Caldwell was born in the Carmel Community of Meriwether County, Georgia, in 1899.
That same year he became a part-time assistant professor of law at Emory University in Atlanta, Georgia, in addition to practicing law at the firm of King, Caldwell and Partridge.
Five years later, Caldwell assumed a professorship at the UGA School of Law.
While Black applicants were told there was no campus accommodation available for them, an internal communication from Caldwell indicated that the university was "relying on this to bar the admission of a Negro girl from Atlanta.
Caldwell Hall, home to the University of Georgia's School of Environment and Design, is named in his honor, as well as the Harmon W. Caldwell Professorship in Constitutional Law in the UGA Law School.