Built in 1800–01, the house was designated a National Historic Landmark in 1970 as a well-preserved Federal style brick townhouse.

The main facade, facing Spring Street, is five bays wide, with a central entrance sheltered by a semicircular portico supported by Doric columns, and topped by a balustrade.

[3] Constructed in 1800-1801 for shipping magnate Major Hugh McLellan, the brick mansion was designed by John Kimball Sr. (1758-1831), an architect/housewright originally from Ipswich, Massachusetts.

The L. D. M. Sweat Memorial Galleries, designed by John Calvin Stevens, were added in 1911 behind the house, to which they connected by corridor.

In 1957, two mantelpieces salvaged from the 1805 Commodore Edward Preble House, designed by Alexander Parris, replaced originals lost during a Greek Revival remodeling of the drawing and dining rooms.