His wife Laura Caroline (1842–1916), daughter of William Russell and Emma Campbell, was a noted beauty in the contemporary society scene.
After his death, in 1899, his wife married Sir Frederick John William Johnstone.
His elder brother was Arthur Egerton, 3rd Earl of Wilton.
His father was Thomas Egerton, 2nd Earl of Wilton (1799–1882) and his mother was Lady Mary Margaret Stanley, daughter of Edward Smith-Stanley, 12th Earl of Derby.
His paternal grandfather was Robert Grosvenor, 1st Marquess of Westminster.