He had begun helping slaves to escape as early as 1839,[2] and late in his life Ashley relished telling stories of the families he had saved as a 17 year old.

[3] He told the story later in life, which came down through the family that, when he left the home at 14, the last words his father said to him as he went off was: "You're on the straight road to Hell, boy!"

Instead, by 1851, abolitionist activities caused Ashley and his wife to flee north to Toledo, Ohio to avoid prosecution under the Fugitive Slave Act of 1850.

There, Ashley opened a drug store (which was soon burned down) and also became involved in the new Republican Party, campaigning for its presidential candidate John C. Fremont and congressman Richard Mott.

However, he opposed the Church of Jesus Christ of Latter-day Saints and especially polygamy, and limited Utah's boundaries to reduce Mormon influence.

"[7] He also criticized Johnson for attempting to veto extensions of the Freedmen's Bureau, the Civil Rights Bill and the Reconstruction Acts.

His political appointments, and support for public education, including of Chinese immigrants, proved unpopular in the Democratic-leaning territory.

He died of heart failure after a fishing trip on September 16, 1896, in Alma, Michigan, and was interred in Woodlawn Cemetery (Toledo, Ohio).

A eulogy at the Unitarian Church in Ann Arbor, Michigan, mentioned his large size, "intellectually, physically and morally.

"[12] Three years before his death, his efforts on behalf of racial equality were recognized by the Afro-American League of Tennessee, and he donated the proceeds of a book of his speeches to build schools.

In early 2010, the Ohio Historical Society proposed Ashley as a finalist in a statewide vote for inclusion in Statuary Hall at the United States Capitol.

Civil rights figure Frederick Douglass regarded Ashley as a white man who had a determination to secure equal justice for all, along with the likes of Benjamin Wade, Thaddeus Stevens, and Charles Sumner.

[13] A contemporary journalist, Benjamin Perley Poore, said Ashley was a "man of the lightest mental calibre and most insufficient capacity" who "passed much of his time in perambulating the aisles of the House, holding short conferences with leading Republicans, and casting frequent glances into the ladies' gallery.