The Andrew Safford House at 13 Washington Square West, built in 1819, was said to be the most expensive home in New England at the time.

The Library is a two-story building, although the second floor is two normal stories high (with suitably large windows), and originally served as exhibition space.

[2] The area behind the Phillips Library and south of Brown Street is a garden area that includes two other historical structures: the Vaughan Doll House, a modest late 17th century one-room structure that may have been a Quaker meeting house, and the Lyle-Tapley Shoe Shop.

To its east, at the corner of Essex and Washington Square West, stands the Bentley-Crowninshield House, a c. 1727 Georgian house that was relocated from a site across Essex Street in 1860.

On December 8, 2017, much to the dismay of Salem residents, Dan L. Monroe, PEM's Rose-Marie and Eijk van Otterloo Director and CEO, issued a press release announcing that the 42,000 linear feet of historical documents will be permanently relocated to Rowley, MA and Plummer Hall and Daland House, the two historic buildings which had housed the Phillips Library, will be utilized as office and meeting space.