The Georgian style "mansion" was first occupied by Chief Justice Frederick Smyth by rent and approval of "The Proprietors" on April 10, 1766, to 1773.

In June 1776, after the onset of hostilities between Great Britain and the fledgling nation, he was ordered arrested and taken to Princeton for trial and then imprisoned in Litchfield, Connecticut until his release in a prisoner exchange in 1778.

Rattoon, a Perth Amboy merchant, had carefully concealed his work as an operator for the British secret service.

Rattone was an ambitious New Yorker who added a large wing and turned the property into The Brighton, a hotel of grand stature in its day.

In subsequent years, the original 11.5-acre site was subdivided, and the former Proprietary House was converted into a Presbyterian retirement home.

In 1808, Rattoon sold the property to Richard M. Woodhull of New York City who converted it into a hotel called The Brighton.

In 1914 a group of Perth Amboy citizens founded the Westminster Historical Society to raise funds to purchase and restore the Proprietary House.

In the late 1930s, the house was measured and plans drawn by the Historical American Buildings Survey for a record of this significant property.

Mainly with volunteer help, the association cleaned out years of accumulated debris and removed walls which had divided the great rooms.

At its cost, the Partnership would restore and renovate the exterior, and finish the interior of the 1809 wing and the upper floors of the main block as offices.

The ground and first floors of the original mansion are leased by the State to the Proprietary House Association, which is charged with raising funds for programming, interpretation, and historically accurate restoration.

Through paint analysis, soon each room on the first floor and basement levels will depict a colorful timeline of its many past occupants.

The Richard McGuire graphic novel Here is set on the grounds of the Proprietary House, primarily in a family home built on one of the land lots sold in 1904.