Originally the family seat of Philipse Manor, and later Yonkers city hall, it is Westchester County's second oldest standing building after the Timothy Knapp House.

Located near the Hudson River at Warburton Avenue and Dock Street, it is owned and operated by the New York State Office of Parks, Recreation and Historic Preservation.

The southwest corner, the oldest part of the structure, was built around 1682 by Dutch-born merchant and trader Frederick Philipse, the first Lord of Philipsburg Manor, and his wife Margaret Hardenbroeck.

[citation needed] Philipse, who by his second marriage became a son-in-law of Stephanus Van Cortlandt, had amassed by the time of his death a 52,000-acre (21,000 ha) estate along the Hudson River that encompassed the entire modern city of Yonkers and much of western and lower Westchester County.

The question became moot when Eva Smith Cochran, matriarch of a wealthy local carpet milling family, stepped in and donated $50,000 to the city as a nominal reimbursement for their care of the building during the previous 40 years.

Represented among the 60 paintings are nearly all of the presidents of the United States from Washington to Calvin Coolidge, as well as war heroes, historical figures, and members of the Philipse family.