His grandfather died in 1829 although his father legally received the Jay Estate in Rye in 1822 (which was later inherited by his elder brother John).
[4] Jay was described by General John Watts de Peyster as "a perfect specimen of the typical French nobility, pure blood; handsome; well made; graceful; easy, agreeable, and as full of elegant wickedness as an egg of meat.
"[5] According to his obituary, "To all Mr. Jay was a courteous gentleman; but only to those who knew him well and rejoiced in his friendship, were shown in their full strength the true and noble traits of his nature.
The truthfulness, sincerity, frankness, which so beautifully adorned his character, were themselves softened and mellowed by a chivalrous generosity.
"[6] On January 13, 1848, Jay was married to Josephine Pearson (1829–1852) of Washington, D.C.[7] Together, they resided in Washington,[6] and were the parents of one child before their early deaths:[8] His wife died on January 3, 1852, at twenty-two years old, just under two weeks before their fourth wedding anniversary.