It is located at 1 Church Street, in an architecturally sophisticated Renaissance Revival structure, built in 1894 as a gift from George Amos Abbott, a local mill owner.

The sloping block is landscaped, with an open lawn raised from Church Street with benches in front of the library, and a small garden in the rear, along with memorials to the town's military.

The library is a T-shaped masonry structure, one story in height, with a brick exterior, granite foundation, and a hipped slate roof.

The main facade, facing west toward Church Street, is five bays wide, with a projecting gabled entry at the center.

The entrance below is set in a tall round-arch and keystoned opening, the arch supported by attached Ionic columns and flanked by floral motifs.