Pelham was the eldest son of Charles Anderson-Pelham, 3rd Earl of Yarborough, and his wife, Lady Victoria Alexandrina Hare, daughter of William Hare, 2nd Earl of Listowel.

[2] When Yarborough inherited his father's titles in 1875, he took up his seat in the Lords as a Liberal but later became a Conservative over Irish Home Rule.

In 1890 he was admitted to the Privy Council[3] and made Captain of the Honourable Corps of Gentlemen-at-Arms under Lord Salisbury,[4] a post he held until 1892.

[5] During the Second Anglo-Boer War a new regiment was formed as the Lincolnshire Imperial Yeomanry, of which Yarborough was appointed Lieutenant-colonel in June 1901[6] After the war it became a permanent unit as the Lincolnshire Yeomanry.

Lord Yarborough was appointed Honorary Colonel of the 3rd (Militia) Battalion of the Lincolnshire Regiment in 1898 and of the 5th Battalion, Lincolnshire Regiment (Territorial Army) in 1922.