Potts was born in Upper Marlboro, Maryland, and lived there until he moved with his family to the Barbados Islands in 1757.

He returned to Maryland and settled in the state capital of Annapolis in 1761, where he studied law.

Potts served as a member of the committee of observation for Frederick County in 1776 and as military aide to the Governor of Maryland in 1777.

He was appointed by President George Washington as United States attorney for Maryland, and served from 1789 to 1791.

He declined the nomination to be elected to the Maryland State Senate in 1787, but served as a presidential elector in 1792.