He was in that office when responsible government was granted to Cape Colony, British Columbia was added to the Dominion of Canada and during the First Boer War.

At the end of 1882 he exchanged this office first for that of Chancellor of the Duchy of Lancaster and then for the secretaryship of state for India, a post he retained during the remainder of Gladstone's tenure of power (1882–1885, 1886, 1892–1894), though in 1892–1894 he combined with it that of the lord presidency of the council.

Sir Edward Grey who served as Parliamentary Under-Secretary under Kimberley at the Foreign Office portrays him unfavourably as prolix and prone to irrelevant digressions in conversation although concise, definite and clear on paper.

[6] However, according to the Encyclopædia Britannica Eleventh Edition, "As leader of the Liberal party in the House of Lords he acted with undeviating dignity, and in opposition, he was a courteous antagonist and a critic of weight and experience".

[3] They had three children: He died at 35 Lowndes Square in London (now the High Commission of Pakistan) on 8 April 1902,[3] aged 76, and was succeeded in his titles by his eldest son, John.