Built in 1880, it was the town's first modern graded school, and is a prominent local example of Colonial Revival architecture.

It was named for a native son, United States Supreme Court Justice Joseph Story, and served as a public school until 1978.

Windows are set in segmented-arch openings with keystones at the tops, and there is a gabled pedimented in the roof above the main entrance, in a section that projects slightly.

In the first half of the 19th century, the town established a series of district schools, including one at the site of this building.

This building, completed in 1880, was the first brick school built by the town, and its first major investment in improved education.