His uncle, also a James Speed, was United States Attorney General in Abraham Lincoln second term Cabinet and for a short time Andrew Johnson's.

He then moved to Chicago and worked in the office of Badger & Co.[1] Speed enlisted with the 1st Ohio Battery and was later made adjutant of the 27th Kentucky Infantry Regiment under Colonel Pennypacker.

[1] He developed and operated large coal interests, and served as president of the Ohio Valley Telephone Company.

[3] As a philanthropist, he donated a statue of Abraham Lincoln, which stands in the State Capitol, and he helped to establish and maintain several schools.

After suffering from Bright's disease (modern-day nephritis) for years and experiencing heart issues, Speed died on July 7, 1912, at Hotel Samoset in Rockland, Maine,[1] where he spent his summers.