A grandson, Isaac E. Avery, served as a colonel in the Confederate States Army during the American Civil War, perishing at the Battle of Gettysburg.
Ellsworth would go on to become a congressman, a framer of the Constitution, and a Justice of the Supreme Court; the two remained fast friends, even after Avery moved to North Carolina following graduation.
He took a leading role, along with other Princeton graduates like Joseph Alexander, Hezekiah Balch, and David Caldwell, in the unsuccessful attempt to win a Royal charter for what would have been North Carolina's first college, in 1771.
Powell, Vol I. p. 70) "In 1780, while occupying Charlotte, Cornwallis ordered the burning of Avery's office; of his books and papers, only those stored at the home of his friend Hezekiah Alexander were saved.
According to legend, Avery, already one of the state's most prominent lawyers by then, would often proclaim "I refer to Bacon"—meaning The Elements of the Common Laws of England, the noted legal text written by Francis Bacon—when making a point.
The Swan Ponds plantation home built by his son Isaac Thomas Avery in 1848, was listed on the National Register of Historic Places in 1973.