In 1820 he was presented by his kinsman, Henry, Third Earl of Bathurst, to the Rectory of Barwick-in-Elmet, Yorkshire, and continued there as rector for thirty-two years.
In 1852 he resigned the rectory because of conscientious scruples in relation to parts of the baptismal and burial services in the Book of Common Prayer.
He retired into private life and first lived at Darley Dale, near Matlock, Derbyshire, where for eleven years he gave himself to literary pursuits.
During his early years of ministry, Bathurst composed hymns and versified a large portion of the psalms.
These were published, 1830, in a small volume entitled Psalms and Hymns for Public and Private Use.