Peter Augustus Jay was born at Liberty Hall", on January 24, 1776, at the home of his maternal grandparents' in Elizabethtown, New Jersey.

[2] His mother was the eldest daughter of 13 children born to New Jersey Governor William Livingston (1723–1790).

[11] Jay shared his father's commitment to social justice and actively pursued greater rights for African Americans.

[12] Jay is best known for giving a speech in 1821 at the New York State Constitutional Convention as a delegate arguing that the right to vote should be extended to free African Americans.

[16] Together, they had eight children, including:[11] His wife died in Madeira, an archipelago in the north Atlantic Ocean, southwest of Portugal, on December 24, 1838.

Under his father's aegis, Peter Augustus installed European styled stone ha-has on the property and planted elm trees.

In 1836, Peter Augustus contracted with a builder, Edwin Bishop, to take down the failing farmhouse that had been barraged by the British during the Revolutionary War.

Reusing structural elements from "The Locusts" where his father grew up as a boy, Peter Augustus Jay helped create the Greek Revival mansion that stands there today.

It is because of this legacy of social justice that the Jay site was added to the Westchester County African American Heritage Trail in 2004.