A granddaughter of Philip Ludwell and Benjamin Harrison II, she was a prominent figure within the American gentry.

She grew up at Green Spring Plantation, her family's home in James City County.

[4] Her paternal grandfather, Philip Ludwell, was the first Governor of the Province of Carolina and served as Speaker of the Virginia House of Burgesses.

Lee's brother, Philip Ludwell III, was the earliest-known Eastern Orthodox Christian in North America after his conversion to Russian Orthodoxy in 1738.

[3][7] She brought a large dowry, as well as social and political prestige, to the marriage, therefore increasing her husband's wealth and status in the colony.

Lee, who was pregnant with her fourth child, was thrown from her room on the second floor in order to escape the flames, resulting in the miscarriage of a son.