The original house was built about 1773 and subsequently enlarged and altered a number of times.

The original structure is a 2+1⁄2-story, five-bay brick house, altered by the Van Cortlandt family in the 1830s.

A large 2+1⁄2-story frame wing covered in stucco was built in the 1920s.

After 1783, the house was occupied by his son Pierre Van Cortlandt, Jr. (1762–1848) until his death.

[2] During the American Revolutionary War, the house served as headquarters for General George Washington from November 10 to 12, 1776; November 28 to 30, 1779; and June 25 to July 2, 1781.