His father, J. DeForest Richards, was a leading Congregational minister and educator who served as President of the Alabama State University in Tuscaloosa.

[2] In 1871, he married Elsie Jane Ingersol, an Alabama native descended from the prominent northeastern family.

Richards relocated to Alabama during post-Civil War Reconstruction, moving with his father in 1865 to run a cotton plantation.

He reorganized the enterprise and worked as a cobbler for two years to repay his debts and accumulate capital to start a new business.

He then opened a merchandising business in Camden, Alabama, where he built a "large and profitable trade," becoming a leader in the community.