Castlemaine did not father any of his wife's children; Charlotte and her siblings were the illegitimate offspring of their mother's royal lover, Charles II.

The diarist Samuel Pepys noted that the child would likely have good marriage prospects: "my Lady Castlemayne [Barbara Villiers] will in merriment say that her daughter (not above a year old or two) will be the first mayde in the Court that will be married…"[3] Charlotte was for her royal blood and connections seen as a great match and in 1672, Sir Francis Radclyffe; an ambitious man tried to bring about a marriage between the eight-year-old Charlotte and his nineteen-year-old son Edward Radclyffe.

Charlotte and her younger sister Barbara were sent to be educated at the convent of the Conceptionist Order of the Immaculate Conception of Our Lady,[8] at Faubourg Saint-Antoine in Paris.

"[13]Villiers was said to be a "cruel and austere" parent would often use her children for emotional blackmail against the king at one time threatening "to bring all of his (Charles) bastards to his closet - door" (i.e., abandoning them) Another evidence for her being a very harsh mother is when Charlotte accidentally broke the glass window of her mother's coach when traveling through the park; she was so distraught that she started crying in fear of her punishment.

[15] Charles also funded a townhouse to be built by Sir Christopher Wren in 1677[16] named Lichfield House, it was adjoined by another home which is today known as 10 Downing Street.

Lee participated in court life and was frequently in presence at Windsor castle[21] and is mentioned by as being present at a gathering on 24 Jan 1682 in honor of the Moroccan ambassador Mohammed ben Hadou.

and among these were the King's natural children, namely, Lady Lichfield...In 1685, her father suffered an apoplectic fit and died four days later at the Palace of Whitehall.

After the death of her father in 1685, Charlotte continued to be part of the inner circle of the new king, who was her uncle James and a great favorite of his wife Mary of Modena.

One year later, Charlotte's younger sister, Barbara, who had been seduced by count of Arran, gave birth to an illegitimate son and was from thereon disowned by their mother and sent to a convent.

On their funeral monument the inscription reads; “at their marriage they were the most grateful bridegroom and the most beautiful bride and that till death they remained the most constant husband and wife.”[27][25]In her will, she left her three younger sons £100 each,[28] and to her daughter Barbara £2000[28] (if she married with the consent of her brother) and £200[26] to the Roman Catholic bishop Bonaventure Giffard.

[28] The French writer Charles de Saint-Evremond included Lee as a topic of interest[29] in his vignette of imagined conversation between a nobleman and Mrs. Myddelton Madam Litchfields more vibrant demeanour Would endear her greatly to all those who meet herThe countess of Lichfield appears as a character in "Devil water"[30] (1961) by Anya Seton.