The property, historically known as Hyde Park, was one of several homes owned by Frederick William Vanderbilt and his wife Louise Holmes Anthony.

The interiors are archetypes of the American Renaissance, blending European architectural salvage, antiques, and fine period reproductions representing an array of historical styles.

[3] The earliest development of the estate began in 1764 when Dr. John Bard purchased land on the east side of the Albany Post Road, where he built Red House and developed the agricultural aspects of the eastern section of the property that continued through Frederick and Louise Vanderbilt's occupancy.

In 1828, Dr. David Hosack, president of the New York Horticultural Society, purchased the property from Samuel Bard's heirs, with André Parmentier helping to design the grounds.

In 1840, John Jacob Astor purchased the property from Hosack's heirs for his daughter Dorothea and her husband Walter S. Langdon.

[3] Charles Follen McKim designed the plan in the Beaux-Arts style and Stanford White assisted as an antiques buyer.

The mansion also included plumbing and forced hot air central heating and electric lighting which was powered by a hydroelectric plant built on the estate on the Crum Elbow stream.

[5] Robertson's coach house, completed in the Queen Anne style, is in vast contrast to stoic mansion.