Lord Southampton was also the great-great-grandson (through an illegitimate line) of King Charles II by his mistress Barbara Palmer, 1st Duchess of Cleveland.

The Southampton title had previously been created for Charles FitzRoy, eldest natural son of Charles II and the Duchess of Cleveland and the elder brother of Henry FitzRoy, 1st Duke of Grafton, but had become extinct in 1774 on the death of his son William FitzRoy, 3rd Duke of Cleveland and 2nd Duke of Southampton, six years before the creation of the barony of Southampton.

Lord Southampton was succeeded by his eldest son, the fourth Baron.

Edward FitzRoy, second son of the third Baron, served as Speaker of the House of Commons from 1928 until his death in 1943.

William FitzRoy (1830–1902), a great-grandson of the first Baron, was a major-general in the army.