He was the son of Robert Kelly (died 1825) who came to New York City from Ireland in 1796, and became a prosperous merchant.
In 1750, the "Ellerslie," land in Rhinebeck, Dutchess County, New York, was the farm of Hendricus Heermance.
The farm passed to the Kips by inheritance, and was in 1814 sold to Maturin Livingston, son-in-law of Gov.
Livingston built a mansion on it, and in 1816 sold the property to James Thompson, who named the estate "Ellerslie."
In 1841, it was sold to William Kelly, who increased the acreage to nearly eight hundred, and greatly beautified the estate.