He married Alabama Pippen on January 18, 1865, and they had five children,[2] Annie Augusta, Ida Lee, James Daniel, Allie Sue, and John Watkins.

[3] Before graduating from college, Richardson enlisted in the Confederate States Army during the American Civil War, and served nearly four years.

Richardson studied law; was admitted to the bar and commenced practice January 1, 1867, in Murfreesboro, Tennessee.

Pursuant to an act of Congress on August 20, 1894, Richardson was charged with compiling the "Messages and Papers of the Presidents," a multi-volume work including every single important document from the federal Government, from the early days of President Washington through the second administration of Grover Cleveland, plus some papers from the administration of William McKinley.

[8]: 262  He also became the eleventh Sovereign Grand Commander of the Scottish Rite's Supreme Council for the Southern Jurisdiction.