By the early 1900s, the family owned 40,000 acres (63 sq mi; 160 km2) in the area, half of it comprising the Arden Estate.

James Earle Fraser did a bas-relief portrait of Harriman over one of the fireplaces, as well as a fountain in the interior court; Malvina Hoffman did a bust of Mrs. Harriman; and Charles Cary Rumsey did a fountain of the Three Graces, a marble fireplace surround that featured a caricature of architect Thomas Hastings, and corbel carvings of bighorn sheep in the music room.

On the second floor was an "Indian Corridor", featuring photographs of Native Americans taken by Edward S. Curtis during the Harriman Alaska Expedition of 1899.

[6] In 1950, Averell Harriman and his brother Roland deeded the property to Columbia University, as "home of The American Assembly", a public policy institution founded by Dwight D. Eisenhower the same year.

The house was identified as America's first conference center, and became a National Historic Landmark in 1966[1][7] but is not open to the public.

[9] The house was purchased in 2011 by a Chinese-backed nonprofit, the Research Center on Natural Conservation, Inc.[10][11] In 2015, the same group bought the bankrupt New York Military Academy.