Because the building sits on a sloping lot, the rear section is actually three full stories; its ground floor section is made of concrete, while the rest of the building is predominantly made of brick, laid in a variation of an English cross bond pattern.

The central portion was built in 1871 to a design by Henry M. Francis, originally with Second Empire styling, and named the Butler School in honor of a long-serving town clerk, Caleb Butler.

Exterior changes also gave the building its present Colonial Revival appearance.

Despite significant alterations in use to accommodate different grades over the years, the building's interior also remained remarkably intact.

This building was renamed the Colonel William Prescott School, and house lower grades until it was closed in 2008.