Construction of the three-story wood-frame First Period house was begun by Captain John Whipple in 1677 at the corner of Saltonstall and Market Streets.

By 1705, a series of 3-sided 1-story timber frame additions (lean-tos) had been added on the west side of the house, including a kitchen (the third chimney), a buttery, and sleeping quarters for the household slaves and servants.

This was a very large house then (and now), with high ceilings in all rooms except the lean-tos, a prominent location near the mills and with land abutting the Ipswich River.

In the 1720s, a daughter who had inherited the house from Major John Whipple altered the facade and some of the interior to reflect the new and fashionable Georgian-style architecture.

The Historical Society restored some of the interior to the assumed 17th-century style and replaced the Georgian windows and exterior doors with reproductions of the 17th-century originals.