The walls of the new house (built of brick on a large ground area) had reached the height of the first story and a great stairway had been placed in the main hall when hostilities halted construction.
A makeshift roof was placed over the walls and the owner retired in deep gloom to this peculiar dwelling, to which the name "The Hermitage" was given.
Henry sold The Hermitage to his younger brother, Peter Schuyler Livingston (1772–1809), who was listed as the owner when it was publicly advertised for sale in 1799.
The property was bought at auction for $19,000 in 1899 by the heirs of the late James B. Jermain of Albany, who included among them Catherine B. J.
"[4] In 1930, The Hermitage was purchased by Ida Helen Ogilvie, founder and former head of Barnard College's Geology department.
During World War II, she was involved with the Woman's Land Army of America to work in agriculture replacing men called up to the military.
"[2]According to writer/director James Ivory (who owned the Jacob Rutsen Van Rensselaer House and Mill Complex in Claverack, New York), he said: "It makes you heartsick.
"[2] As part of the Historic American Buildings Survey, The Hermitage was photographed by Nelson E. Baldwin on January 16, 1937.