When the British took over the Dutch colony in 1674, Philipse pledged his allegiance to the Crown and was rewarded with a title and manorship for his holdings, which ultimately grew to some 81 sq mi (210 km2).

He owned the vast stretch of land spanning from Spuyten Duyvil Creek in the Bronx (then in lower Westchester County), to the Croton River.

The land that would become Philipsburg Manor was first bought from Adriaen van der Donck, who had invested in an unsuccessful Dutch patroonship in New Netherland before the English takeover in 1664.

Philipse made several additional purchases between 1680 and 1686 from the Wiechquaeskeck and Sintsink Indian tribes, expanding the property to both the north and south; he also bought a small plot of land from the Tappans west of the Hudson River.[where?]

Philipse also bought out his partners' stakes during this time,[4] enticing friends from New Amsterdam and Long Island to move with him with the promise of free land and limited taxes.

Philipse built a simple residence in today's Getty Square neighborhood of Yonkers, New York near the confluence of the Nepperhan River with the Hudson.

In 1697 Adolphus Philipse, Frederick's second son, purchased a tract from Dutch traders which received British Royal sanction as the Highland Patent.

By the terms of Frederick Philipse's last will and testament, dated 26 October 1700, proved 1702, Adolphus received all the Manor north of Dobb's Ferry, including the present town.

[21] Frederick III leased the entirety of his property to William Pugsley before siding with the British in the American Revolution and leaving New York City for England in 1783.

After the Revolution, the entire Philipse holdings, including the Manor and other lands in today's Westchester County, and the Highland Patent, were seized by New York and sold by its Commissioners of Forfeitures.