Born on October 28, 1868, in Gordonville, Alabama, Caffey received an Artium Magister degree in 1887 from Howard College (now Samford University), an Artium Baccalaureus degree in 1891 from Harvard University, an Artium Magister degree in 1892 from the same institution and attended Harvard Law School.

[1] Caffey briefly gained notoriety when his name was associated with possible political advocacy on the part of the Department of Justice.

Just two weeks before the 1920 election, John R. Rathom, publisher of the Providence Journal, charged that the Democratic candidate for Vice President, Franklin D. Roosevelt, had acted improperly while Assistant Secretary of the Navy in releasing sailors convicted on morals charges from Portsmouth Naval Prison.

Caffey, once the Attorney General had telegraphed his authorization, released a lengthy document from Justice Department files that discredited Rathom.

A report in the New York Times suggested that he may have acted with undue enthusiasm: "Some surprise was shown at the [D]epartment [of Justice]...that District Attorney Caffey had given the entire Rathom letter to the newspapers.