Work on the estate began in 1903, when John Shaffer Phipps promised his fiancée, Margarita Grace (a daughter of businessman Michael P. Grace), that he would build her a home in the United States that resembled her family's British residence at Battle Abbey in Battle, East Sussex, England.

The painting of Mrs. Henry Phipps and Her Grandson Winston (1906–07) by John Singer Sargent hangs in the dining room.

Julie Newmar (1995), Cruel Intentions and 8MM (both 1999), The Manchurian Candidate (2004), Hitch (2005), Kabhi Alvida Naa Kehna (2006), Bernard and Doris (2006), American Gangster (2007) and Arthur (2011).

It served as the inspiration for the Buchanan Estate featured in Baz Luhrmann's 2013 film adaption of The Great Gatsby by his wife and production designer Catherine Martin.

In August 2024, a grant was released by The Preservation League of NYS to support the fund noteworthy restoration project of the garden.