[a] Both his parents, Thomas Lee and Hannah Harrison Ludwell were of the First Families of Virginia, with considerable wealth as well as political power.
[4]: 49, 65–68 In 1763 he married his heiress ward, Elizabeth, daughter of James Steptoe of Westmoreland County, who survived him and remarried to Philip Richard Fendall,[1]: 72 but their son died as an infant.
[1]: 72 Although nicknamed "Colonel Phil" for his rank in the local militia, Lee's primary business was operating Stratford Hall and associated plantations he inherited mostly pursuant to primogeniture (including more than 12,000 acres in what had been Northumberland and Stafford Counties in Virginia).
His property in Dorchester County, Maryland, on the northwest fork of the Nanticoke River had been part of land owned by Capt.
Although he attributed the delays to needing to settle debts incurred by their father, they resented his lavish entertainments at Stratford Hall.
In 1787 his sister Letitia Corbin Lee of Harford County sold 200 acres called "Rehoboth" to John Smoot.