Captain John Coffeen, the town's first permanent settler, brought his family and possessions into the wilderness of Cavendish in June 1769.

They built a dwelling in the northern part of town on what is now E. I. Heald's farm, on the lot still called the "Coffeen pasture".

In the early 1780s, Leonard Proctor and Salmon Dutton came from Massachusetts and gave their names to the two major settlements on the Black River, Proctorsville and Duttonsville.

He is credited with having conducted a 1784 survey for the first road from Duttonsville along the Black River to Ludlow (now Vermont routes 103 and 131).

The marriage of Redfield Proctor and Emily Dutton in 1858 joined the leading families of the two villages and promised to put an end to the former rivalry.

[8] Cavendish was one of thirteen Vermont towns isolated by flooding caused by Hurricane Irene in 2011.