First granted in 1733 as "Lower Ashuelot", Swanzey was one of the fort towns established by Governor Jonathan Belcher of Massachusetts, which then claimed the territory.

The town was the site of many battles during King George's War, and in 1747 it was abandoned for three years.

It was regranted to 62 proprietors on July 2, 1753, by Governor Benning Wentworth as "Swanzey", most probably because some early settlers had a connection to Swansea in Wales.

[4] The town features four covered bridges, and was the home of theatrical trouper Denman Thompson, who gained a national reputation by his portrayal of the Yankee farmer, "Joshua Whitcomb", star of his stage play The Old Homestead.

Residents restage Thompson's melodrama every summer at a natural outdoor amphitheater called the Potash Bowl.

The town's highest point is the summit of Franklin Mountain at 1,423 feet (434 m), on the border with Winchester.