He was the eldest son of Neil Primrose, 3rd Earl of Rosebery and, his second wife, Mary Vincent.
[4] His paternal grandparents were James Primrose, 2nd Earl of Rosebery and Mary Campbell (a daughter of the Hon.
His maternal grandparents were Sir Francis Vincent, 7th Baronet and Mary Howard (the daughter of Lt.-Gen. Hon.
Before their divorce in 1815, they had four children:[4] Lady Roseberry caused a society scandal when she had an affair with her brother-in-law, Sir Henry St John-Mildmay, 4th Baronet (the widower of her deceased sister Charlotte).
She divorced Lord Rosebery and married St John-Mildmay in Stuttgart after obtaining a special permission by the King of Württemberg.