Built about 1807, it is one of the city's oldest surviving stone buildings, further notable for a parade of locally or statewide prominent residents.

It is 2+1⁄2 stories in height, built out of ashlar cut red sandstone that is stuccoed except for the corner quoining blocks, and covered by a gabled roof.

Windows are rectangular sash, arranged symmetrically around a center entrance in openings with stone sills and lintels.

Goodrich had the building's third floor ballroom built; this work is attributed to a locally prominent master builder, David Hoadley.

In 1858 the house was purchased by Charles Atwater, another prominent businessman and politician who served in the state legislature and was a candidate for Governor of Connecticut.