William was born in Somerset County to John and Betsy (Murray) Stone.

Bishop Thomas Claggett ordained him a deacon in Prince George's County on December 3, 1803.

After his ordination as priest, Kemp became rector of Stepney Parish then in Somerset County, Maryland.

He served there for over twenty years until he was transferred to be rector of St. Paul's in Chestertown, Maryland in 1829.

He died in 1838 at Salisbury, Maryland and is buried next to his wife Anne Savage Stone (d. 1821) at the Parsons Cemetery in Wicomico County.