The Whiting House is prominently sited at the northwest corner of the junction of Main and High Streets, on the fringe of Ellsworth's central business district.

It is roughly square in shape, with a slightly projecting central pavilion in the front (south-facing) facade that is topped by a mansarded observatory.

The entrance is sheltered by a flat-roof portico supported by clusters of columns, with low balustrades to the sides.

The building's corners are decorated with paneled pilasters, and there is a projecting single-story bay to the east.

[2] The house was commissioned by Samuel Kidder Whiting, a local dry goods merchant, in 1871, when he was 54 years old, and it was his home until his death in 1917.