[5] Charles was the third of eleven children born to Henry (1730–1787) and Lucy (Grymes) Lee on his father's Leesylvania plantation in Prince William County, Virginia.

Another lawyer brother became Congressman Richard Bland Lee, and future President Zachary Taylor would be his third cousin.

[6][7] His father's 1787 will appointed Charles as Executor and the guardian of his three siblings under 21 years of age (Edmund, Lucy, Ann).

In addition to his government duties described below Lee had a private legal practice, and one of his clients was George Washington (from 1785 until his appointment as Attorney General as described below in 1795).

Like his uncle squire Richard Lee, Charles Lee also held appointed positions as (most lucratively) tax collector for the Port of Alexandria (1789–1793), naval officer for the South Potomac (1777–1789), secretary of the Potomac Company (1785), and clerk for the Common Council of Alexandria (1785).

[14] President George Washington appointed Lee the Attorney General after William Bradford died in office.

In 1804 fellow councilors elected Lee as Alexandria's mayor, but he refused to serve, so Dr. Elisha C. Dick was selected instead.