Gregg was born on October 8, 1819, in Society Hill, South Carolina, Darlington County, South Carolina, in an area historically known as "the old Cheraws", the son of David Gregg and Athalinda Brocky.

He was ordained deacon on June 10, 1846, and priest on December 19, 1847, at St Philip's Church by Bishop Christopher Edwards Gadsden of South Carolina.

He spent his whole ministry, just before his election as bishop, as the rector of St David's Church in Cheraw, South Carolina.

Bishop Gregg presided through the difficult days of the Civil War and Reconstruction, and gave leadership as Texas changed from frontier to settled community.

Bishop Gregg died in office at his Austin, Texas, home on July 10, 1893, is buried at Saint David's Church in Cheraw, South Carolina.