The Swann's Point area, located west of the mouth of Gray Creek, has a rich historic of precolonial Native American occupation, as well as significant early colonial settlements.

[5] It was here, then, that petitions were heard complaining of excesses during the rebellion's suppression, including by William Hartwell, captain of Governor Berkeley's guard.

Samuel Swann inherited the property from his father, and represented the area in the House of Burgesses several times, but ultimately moved to North Carolina.

During the American Civil War, it was shelled by federal troops and later held an important telegraph line often cut by Confederates.

Later, Gray, who was one of the wealthiest men in Virginia and owned tens of thousands of acres of land bet the Swann's Point property on the game of cards and lost.