He grew up in the nearby town of Lee, where he began his criminal activities at the age of 14 with thefts of "apples, pears, cucumbers, and other fruits of the earth," and then later "a paper money bill" of a neighbor.
Tufts was first imprisoned in 1770, where he attempted his first of many escapes by using the cell's heating fire to burn through a wooden wall of the jail.
When imprisoned, Tufts often saw himself as the unfair victim, once commenting on his time in a jail as "in the shocking circumstances…described, I continued for upwards of three months, without aid or assistance from either friend or foe, or so much as the expectation of relief – no eye had pity on me!"
Tufts spent several years among the Abenaki Indians around Bethel, Maine and learned their natural medicines from Molly Ockett shortly before his final arrest in 1794 at Marblehead, Massachusetts.
After five years imprisoned on Castle Island in Boston Harbor, he was transferred to the jail in Salem, before escaping again to Maine.