The main library is housed in a two-story brick building built in 1853 which is now listed on the National Register of Historic Places.

[5] The cornerstone was placed on August 20, 1853 by the Honorable Abbott Lawrence, who was a friend of George Peabody and served as the ambassador to England.

It is a two-story masonry structure, built out of red brick and brownstone trim and set on a high granite foundation.

The ground floor level has brownstone quoining, rising to a shallow cornice that extends across the facade even with the portico sheltering the main entrance.

[9] At that meeting Eliza discussed making a donation of $20,000 to the library to build a reference room that would be named after her late son, Eben Dale Sutton, who had died in 1862 at the age of 14.

Eliza Sutton sent a letter dated April 7, 1869 to the Peabody Institute Trustees stating her intention to donate $20,000 to build a reference room and a tower, and was to be named the Eben Dale Sutton Reference Room.

[5] Eliza Sutton donated the three volume set of John James Audubon's book Birds of America.

[9] Shortly after the room opened two brass statuettes were used as gas burners, which were to represent Boston and New York.

[9] They were: When Dorothy Cooper retired in 1973 the Sutton Room was closed to the public, and would remain so until 2002 when Martha Holden, the director at the time, hired Nancy C. Barthelemy as the first archivist.

The front portico of the library's entrance was draped in black with the American and the flag of Great Britain.

[14] Four federal soldiers stood guard at all hours and they set up camp in the Sutton Room.

The Peabody Institute opened the South Branch Library on November 27, 1967, and is located at 78 Lynn Street.

[5] The dedication of the South Branch would be attended by Mayor Edward Meaney and Nicholas Mavroules mayor-elect.

[5] The renovation allowed the South Branch to have a new community room, expanded children's area, and new interiors.

Next door to the fire station was a small building that belonged to the West Peabody Community Club.

The new West Branch would be able to have more books and include a large programming/community space and have room for patrons to sit and browse.

The dedication of the West Branch took place on March 11, 1990 speakers include Mayor Peter Torigian; Peabody Institute library director Dr. Mary Ann Tricarico; and trustee Albert Cohen among others.