He was the second son of Colonel Epes Sargent, by his second wife, the widow Catharine Browne.
Among his first cousins was Dudley Saltonstall, a notorious Revolutionary War naval commander.
Through his brother Winthrop, he was uncle to Winthrop Sargent (1753–1820), a major in the Continental Army who was appointed the first Governor of the Mississippi Territory by president John Adams, and Judith Sargent Murray, an early American advocate for women's rights, essayist, playwright, poet, and letter writer.
[5] In 1784, Sargent married widow Margaret (née Whitney) Barnard in Boston.
Together, they were the parents of three sons, all of whom also served in the Assembly, and a daughter, born in Nova Scotia:[5] Sargent died in Barrington at the age of 73.