In addition to its great age, it is notable for its association with Thomas Macy, an early settler of Nantucket and the subject of a poem by John Greenleaf Whittier, and for its long association with the locally significant Colby family.

He left Amesbury in 1659 after years of conflict with local Puritan leaders, most notably over giving shelter to Quakers in this house.

Macy became the subject of a poem by the 19th-century poet John Greenleaf Whittier entitled "The Exile", depicting the plight of Quakers in the religiously intolerant Puritan society of colonial Massachusetts.

He was active in town affairs, served in various offices, and was part owner of a local sawmill.

The property is maintained by the Friends of the Macy–Colby House Association, and is open to the public on Saturdays during the summer.