In the spring of 1767, some of the town's first settlers, Amos Bird and Noah Lee, arrived in Castleton from Salisbury, Connecticut.

Castleton's favorite landmark, Birdseye Mountain, is named for Colonel Amos Bird.

He had acquired 40 shares of land when the town was chartered and built a permanent residence there in the summer of 1769.

[citation needed] In May 1775 Ethan Allen and his Green Mountain Boys met in Castleton with Benedict Arnold to plan their next day's attack on Fort Ticonderoga, 30 miles (48 km) west, on the New York side of Lake Champlain.

Their successful capture of the fort was a holding action that lasted two years until the British launched a powerful sweep southward on Lake Champlain.

A trolley system ran from the center of town to Lake Bomoseen, a destination for tourists vacationing during the summer.

Despite this destruction of hotels and the original commercial and industrial areas of its villages, the town of Castleton retains an architectural heritage spanning two hundred years of Vermont history.

Castleton's mile-long tree-shaded Main Street, with its array of Federal and Greek Revival style houses and public buildings, many by builder Thomas Royal Dake, has been listed almost in its entirety on the National Register of Historic Places.

The house is now maintained by the Castleton Historical Society, and was built and lived in by the Higley family until 1973.

[9] The Cobbler's Shop is an old brick building on the corner of Main Street and Cemetery Road.

It was built at some point between 1774 and 1794 by Nehemiah Hoyt, some people believe it to be the oldest brick building in Vermont.

[10] Castleton is part of the Slate Valley Modified Unified School District.

Lake Bomoseen is the second area, a 5-mile-long (8.0 km) resort and fishing spot in the northwest part of the town.

The third post office is in Hydeville, an extension of Main Street at the south end of Lake Bomoseen.