Thomas "Scotch Tom" Nelson (1677–1747) was a businessman and politician who immigrated from England to become a merchant at Yorktown in the Colony of Virginia.

[1] Arriving at the beginning of the 18th century, he was the immigrant ancestor of the Nelsons, one of the First Families of Virginia.

His son William Nelson (1711–1772) inherited the family business and acquired extensive land holdings throughout the colony.

His grandson Thomas Nelson Jr, a planter and politician, future governor of Virginia, was living there during the Revolutionary War.

[4][failed verification] The site of the Nelson House had originally been settled after the 1620s by immigrant Nicolas Martiau, another ancestor of Governor Thomas Nelson Jr.[5] After being purchased in 1914 by Charles Blow and his wife, the grounds were designed by Charles F. Gillette in 1915.