Charles Palmer, later "FitzRoy", was born on 18 June 1662, and was initially claimed by Roger Palmer, 1st Earl of Castlemaine, his mother's husband, as his son and heir before being publicly acknowledged and invested by King Charles II of England as his son.

His birth marked the separation of his legal parents; Lord Castlemaine, a Roman Catholic, had him christened in the Roman Catholic faith, but six days later, the King had him re-christened in the Church of England as an Anglican and Protestant.

Following the death of her father, the Duchess of Cleveland more or less abducted Mary, with the intention of bringing her up with her own children.

[3] In 1675, he was created Duke of Southampton by King Charles II, along with the subsidiary titles of Earl of Chichester and Baron Newbury.

[4] He was succeeded by his eldest son William FitzRoy (1698–1774), who died without issue, when all his titles became extinct.