Although an English-deeded tract, it is listed by some sources with the patroonships of New Netherland since it incorporated part of that previously owned by Dutch Jonkheer Adriaen van der Donck.

The manor dates from 1693, when wealthy Province of New York merchant Frederick Philipse was granted a charter for 52,000 acres (21,000 ha) along the Hudson River by the British Crown.

He built a facility at the confluence of the Pocantico and Hudson Rivers as a provisioning depot for the family Atlantic sea trade and as headquarters for a worldwide shipping operation.

For more than thirty years, Frederick and his wife Margaret, and later his son Adolph shipped hundreds of African men, women, and children as slaves across the Atlantic.

In 2016, historic restoration work sponsored by the New York State Council on the Arts was completed on the grist mill to rebuild the entire wooden waterwheel and flume.