During the 18th century, it was the home of Henry Lee II, his family and numerous slaves, and known for its productive land and especially the quality of its tobacco.

[5] The original tract of land comprised four headright grants of 50 acres each, given for "each person transported into the 'Kingdom of Virginia'".

[8][9] Around 1750 he chose the site on a high ridge overlooking the Potomac River, which is believed to have been similar to nearby Rippon Lodge, which was built around the same time.

He conspicuously excluded his eldest son, "Light-Horse Harry", from inheriting most of his other properties, and any other belongings, including slaves, likely due to his history of poor financial decisions.

A corner of the house is reconstructed in the Visitors Center, using sandstone from the property and bricks salvaged from the original site.