Born of free parents in Leesburg, Virginia on or about November 9, 1815, he was orphaned about age 10 and sent to live with an uncle; unhappy there, he left and eventually headed to Washington, D.C.[1][2] Grimes was fortunate to grow up a free man, but because he was of mixed race, he identified as African American; witnessing the horrors of slavery in the south, he devoted his life to assisting fugitive slaves and advocating for abolition.

[4] In 1839, Grimes was caught attempting to rescue a family of slaves from Virginia, and he was sentenced to two years in jail in Richmond.

[6] In jail he found religion and after his release in 1840, Grimes was baptized in the Baptist faith and was licensed to preach by a panel chaired by the president of Columbian College, a Baptist institution in the District of Columbia (now the George Washington University).

"[4] He became an important figure in national church organizations and at the American Baptist Missionary Society Convention at Philadelphia in 1858 he, along with Theodore Doughty Miller, William Spellman, and Sampson White, pushed the organization to oppose slavery.

[1] Anthony Burns was an escaped slave from Virginia who came to Boston and became a member of Grimes's church in 1854.

[14] The National Park Service has designated the Loudoun County, Virginia, Courthouse as an Underground Railroad Network to Freedom site in part because of Grimes's trial and conviction there.