The Armstrongs originally called their home "La Bergerie," French for "the sheepfold," as they were raising a large herd of Merino sheep which had been a gift from Napoleon Bonaparte.

There was originally a second straight staircase that led from grade to an elaborate door with sidelights on the second floor which was open to the main hall.

In the mid-19th century William Backhouse Astor enlarged the house from 20 rooms to 48,[6] in brick with brownstone trim, with a semi-octagonal tower on the west side, a north wing, and a third floor throughout the building.

The spectacular Gothic Revival library contained within the tower is reminiscent of the work of Alexander Jackson Davis.

Architect and Chanler family friend Stanford White was hired to enlarge the west drawing room and to install the clerestory in 1895.

[8] The landscaping was improved about 1846 by Hans Jacob Ehlers, who removed a nearby hill to permit a view of the Hudson River.

[5] The property also includes a pair of clapboarded wood-frame barns, additional stables (built about 1850 and destroyed by fire), greenhouse (converted to a garage in 1910, then to a residence in 1965), the square brick gardener's cottage, and a 1+1⁄2-story gatehouse.

John Winthrop Chanler's will provided $20,000 a year for each child for life (equivalent to $470,563 in 2018 dollars), enough to live comfortably by the standards of the time.

[10] They included: As the eldest son, John Armstrong Chanler inherited the property with all its stock, books, pictures, furniture, and personal property, on his twenty-first birthday in 1883, along with $100,000 (equivalent to $2,352,813 in 2018 dollars) for its maintenance,[18] however after his marriage began to disintegrate, he moved to Roanoke Rapids, North Carolina.

[23] Rokeby was the site of an earlier 2007 piece by Kjartansson, titled The Blossoming Trees Performance, during which he recorded himself as a plein air painter for two days.