He served as a Baptist minister in Providence, Rhode Island; Indianapolis, Indiana; Philadelphia, Pennsylvania; and New York City.

[3] His mother Clarissa Roe, of Scotch descent, was thrown from a carriage and killed when James was not quite five months old.

His brother Edward, eleven years older than Simmons, was a teacher in a classic school of Sheffield.

[6] He worked as a farmer and a teacher while receiving his education and also attended prayer meetings.

[2][9] Simmons met Mary Eliza Stevens when he attended Brown University.

[11] After receiving his degree from Brown, and while studying at the seminary, he was a pastor of the Third Baptist Church in Providence, Rhode Island from 1851 to 1854.

[2] Simmons witnessed a fugitive slave named West get shot by a deputy marshal and subsequently was captured.

[12] This led him to deliver a sermon entitled The American Slave System Tried by the Golden Rule and he vowed to act more forthrightly about his beliefs going forward.

After he preached that all men are created equal and called out the governor, the church was set on fire.

[12] He wrote The Cause and Cure of the Rebellion: How far the people of the loyal states are responsible for the war.

However, there were no constructs to help build successful lives, like education and opportunities to move out of poverty.

[16] After working as a minister from 1874 to around 1882, the board of the American Baptist Publication Society elected Simmons as field secretary for the State of New York.