The house was built between 1747 and 1751 on 33 acres (13 ha) in Roxbury by William Shirley (1694–1771), Royal Governor of the Province of Massachusetts Bay, and served as his summer home.

The house is attributed to architect Peter Harrison, and built in part by Thomas Dawes,[4][5] it is one of four remaining mansions of Royal Governors in the United States.

In 1763 Shirley's son-in-law Eliakim Hutchinson, Chief Justice of the Court of Common Pleas for Suffolk County, and one of Boston's richest men, acquired the house from his father-in-law.

After Hutchinson died in 1775, the house was occupied by Colonel Asa Whitcomb's Massachusetts Sixth Foot Regiment, and in 1778 it was seized as Loyalist property.

It then sat unoccupied until purchased in 1782, then passed through a succession of owners, including the widow of a French planter in Haiti, a real estate speculator, and the China merchant James Magee, until it was acquired by Congressman William Eustis, Secretary of War under President James Madison during the War of 1812, Ambassador to the Netherlands 1815–1818, and the first Democratic-Republican governor of Massachusetts from 1823 to 1825.