The sprawling house has sections built as early as 1732, with significant alterations made in 1795 to a design by Asher Benjamin for Oliver Phelps, a major land speculator.

It is a sprawling multi-section wood-frame structure, with a 2+1⁄2-story five-bay central block flanked on the north by a 2+1⁄2-story three-bay section and on the left by a 1+1⁄2-story ell.

The center block, built in 1762, has a large central chimney, and its interior is finished with high-quality Georgian woodwork.

[2] The main block of the house was built about 1762, probably by Abraham Burbank, and was originally more plainly decorated with a gable roof.

Phelps lived here until 1802, when he moved to Canandaigua, New York, to more closely oversee the development and sale of his holdings.