The Romanesque Revival building was constructed in 1884 with funds donated by John Fox Slater, a New England industrialist and philanthropist.

The original building was designed by Stephen C. Earle, and was nearly doubled in size in 1930 with the addition of the Fanning Annex, designed by the Norwich firm of Cudworth & Thompson, and funded by a bequest from local businessman David Hale Fanning.

[1] The library is a 1+1⁄2-story structure, built out of granite with brownstone trim, all finished with a rough quarry exterior.

The 1930 addition nearly doubles the size of the original building, and creates an L shape at the rear; it is built using similar materials.

[4] The museum's displays illustrate life in New England during the 18th and 19th centuries, and are themed around a farmer's workshop, a kitchen, school and children's playthings.