By 1843, the population of the village had grown enough that an association was formed to construct a building to serve the community as both an academy and a church.

A short while later, it was moved to its present location, and remodeled to use for prayer and social meetings on the first floor, and theatrical performances on the upper.

[3] The building was refurbished in the early 1900s, enlarging the kitchen and turning the upper floor into a dining room.

In 1965, the Vermontville Historical Society was organized, and in 1966 received permission from the First Congregational Church, which still owned the building, to convert it into a museum.

[5] The Vermontville Academy is a rectangular two-story gable-roof structure clad with clapboards on a low foundation.