The land, spanning 8,000 acres on Shelter Island, was acquired by English-born colonist Nathaniel Sylvester in the 17th century.

[3] Sylvester and his brother owned two plantations in Barbados and over 200 enslaved Africans.

[5] After her death, he married her sister, Phoebe Dayton Gardiner, with whom he had a fifth daughter, Cornelia Horsford.

[4] With his uncle, Eben Fiske Ostby, he co-founded the Sylvester Manor Educational Farm[7] with the help of the Peconic Land Trust.

[4][8] The manor has been listed on the National Register of Historic Places since April 28, 2015.