His father served briefly in the Virginia House of Delegates when James was growing up, but his main business was as a printer (and he would become half-owner of the Richmond Dispatch in late 1865).

Henry Ellyson also served as sheriff of Henrico County beginning in 1857 and through the American Civil War, and published articles supporting slavery before the election of President Abraham Lincoln, which he had opposed.

[2] James Ellyson attended Columbian College in Washington, D.C. for one term after the war, reportedly wearing his Confederate uniform.

He then transferred to Richmond College (which his father was helping to restart) for one term, and from which his remaining younger brother William Ellyson would graduate in 1870.

James Taylor Ellyson however, moved to Charlottesville to attend the University of Virginia, from which he graduated with a degree in law after editing the campus magazine and participating in the Sigma Chi fraternity and Jefferson Literary Society in his spare time.

Although the younger Ellyson held progressive views concerning labor unions, he was a conservative on race and other matters during what would become his three terms as mayor.

After Ellyson's summertime election nearly four years later, additional black and white Republicans were soon removed from city offices and jobs.