Withers' design, a red brick house of two and a half storeys and raised basement, features many Romanesque Revival touches, such as a recessed front porch with two round-headed arches divided by a spiral column with molded floral design and Corinthian capital.

Other ornaments include an oriel window on the second story, pentagonal dormer on the third, and a parapet roofline.

The fireplace, brick chimney, glazed tiles and oak woodwork are especially well-preserved examples of late 19th-century decor.

[2] The house is the city's most distinguished building in the Romanesque style, complemented nearby by the similar New York State Armory and Harlow Row.

[2] In the later years of the 20th century the house became the home of the United Way of the Dutchess-Orange Region which named it after Samuels, a former benefactor.