While in the national capital and New York City, Holly met several prominent abolitionists, including Frederick Douglass, and by 1848 was working with Lewis Tappan.

The young family soon moved to Windsor, Ontario, across from Detroit, where Holly helped Henry Bibb as associate editor of the Voice of the Fugitive, a weekly paper.

Ordered a deacon on 17 June 1855 at St. Mathew's Church in Detroit, he was ordained a priest on 2 January 1856 in New Haven, Connecticut.

Meanwhile, Holly co-founded the Protestant Episcopal Society for Promoting the Extension of the Church Among Colored People, which worked to have the General Convention adopt a position against slavery and eventually became the Union of Black Episcopalians.

Holly served as rector of St. Luke's Church, New Haven, Connecticut, from 1856 until 1861, during which time he made several trips to Haiti, the world's first black republic.

In 1857, he published a series of lectures as Vindication of the Capacity of the Negro Race for Self Governance and Civilized Progress and in 1859 he lobbied Congressman Frank Blair for funds to establish the emigrant colony.

In 1878, Holly traveled to England as a delegate to the Lambeth Conference, but spent most of the rest of his life within his diocese, on the island of Hispaniola.

These events plunged the island of Hispaniola into another round of civil war, and eventually led to the United States occupation of the Dominican Republic (1916–24).

The liturgical calendar of the Episcopal Church in the United States of America remembers Holly with an annual feast day[8] (Lesser Feast) on March 13,[9] the anniversary of his death, although some churches translate the celebration of his life and accomplishments to November 8, the anniversary of his ordination as well as less affected by Lenten observances.