The house was built by Jacob Rutsen Van Rensselaer, c. 1805, on a portion of what had been the family's 875,000-acre New York estate near Poughkeepsie.

[6] In the latter part of the 19th century, Clifford Miller and his family, bought the home and were responsible for the 1920s addition where they held dances, as well as creating the vast gardens, fish ponds, and landscaping the property with specimen trees.

[4] In 1975, film director and Academy Award-winning screenwriter James Ivory bought the home and 12 remaining acres for about $105,000.

[5] Since acquiring the home, Ivory has painstakingly restored it,[2] guided by Jeremiah Rusconi, a Hudson Valley historical expert (and a BAFTA nominee).

[4] Ivory hosted several actors from his films at the home, including Christopher Reeve, Madeleine Potter, Julian Sands, Helena Bonham Carter, Gwyneth Paltrow, Thandie Newton, Wallace Shawn, Vanessa Redgrave, Rachel Kempson, Eartha Kitt, and Madhur Jaffrey.