Upon returning to Alexandria, Lee began his own law practice and eventually became one of the most renowned and erudite members of the bar.

Besides his public service and legal careers, Lee also worked for his community's religious, educational, and social welfare.

Disturbed by the blight of slavery and its deleterious effects, Lee became an early member of the American Colonization Society.

They had nine children: Edmund Jennings Lee, Jr. (1797–1877), Ann Harriotte (1799–1863), Sarah (1801–1879) known as Sally, William Fitzhugh (1804–1837), Hannah (1806–1872), Cassius Francis (b.

[4] The twilight years of Lee's life were spent quietly at the Lee-Fendall House, where he lived with his daughter Sally and continued to attend to vestry and family business.

Economic depressions in 1815, 1819, and 1837, coupled with the failure of the Bank of the United States and some of his brother's debts, caused him to mortgage his house.

Three years later his son, Edmund Jennings Lee, Jr. recovered the house, and allowed his father, mother, and sister to live in the old family home.