The house, built in 1724,[2] may incorporate in its structure the 1659 hunting lodge of colonial governor Richard Bellingham, and is the only surviving 18th-century building in the city.

It is a two-story wood-frame structure, with a truncated hip roof, two interior chimneys, and a clapboarded exterior.

Its main facade is five bays wide, with a center entrance flanked by Doric pilasters and topped by a transom window, entablature, and gabled pediment.

[4] After his death, the quarter of his land that included the house eventually passed to the Cary family in 1741.

The house remained in the family until 1914, when a local non-profit organization was established to preserve it.