Stratford Hall is also the birthplace of Robert E. Lee (1807–1870), who was General in Chief of the Armies of the Confederate States during the American Civil War (1861–1865).

Designed by an unknown architect, the brick Great House is a two-story H-shaped structure, surrounded on four corners by attending outbuildings, all of which still stand today.

Following the construction of the Great House, Thomas Lee expanded the site into a bustling hive of activity, and soon the working plantation became "a towne in itself" as one visitor to Stratford marveled.

[4] A wharf on the Potomac River was the destination for a large number of merchant ships, a grist mill ground wheat and corn there, and enslaved and indentured servants farmed tobacco and other crops on the thousands of acres of farmland.

Amid this busy world, Thomas Lee and his wife Hannah Harrison Ludwell (1701–1749) raised eight children, six sons, and two daughters.

Thomas Ludwell Lee, active in local politics, served as a Virginian legislator and helped compose the Virginia Declaration of Rights.

Philip Ludwell Lee Sr. (1727–1775), a member of the House of Burgesses and the King's Council, continued to expand the plantation after he inherited Stratford until it eventually encompassed almost 6,600 acres (27 km2).

An arrangement was reached in 1784-1785 that the Fendalls would turn over their rights to Stratford Hall, and Henry would sell a one-half acre lot on Oronoco Street in Alexandria, Virginia for 300 pounds.

Light Horse Harry fell heavily into debt and eventually served a two-year term in debtors' prison.

Stratford Hall passed into the hands of Harry and Matilda's surviving son, Major Henry Lee IV "Black Horse" (1787–1837).