The son of slaveholders, Dillard was educated at Washington and Lee University and held a variety of teaching positions.

He then spent another five years overseeing Norfolk Academy, then was accepted as principal of the Mary Institute at Washington University in St. Louis, in Missouri.

Son Hardy Cross Dillard served as a judge on the International Court of Justice from 1970 to 1979.

[1][3] The James H. Dillard House in New Orleans is today recognized as a National Historic Landmark.

Dillard was appointed a professor of Latin at Tulane University in New Orleans, in 1891; then was named dean of the College of Arts and Sciences three years later.