In 1751 he became Secretary of State for the Southern Department, transferring in 1754 to the Northern Department, and he remained in office until March 1761, when he was dismissed by King George III in favour of Lord Bute, although he had largely been a cipher in that position to the stronger personalities of his colleagues, successively the Duke of Newcastle, Thomas Robinson, Henry Fox, and William Pitt the Elder.

From 1771 to 1776 he acted as governor to two of the King's sons, a solemn phantom as Horace Walpole calls him.

He left no sons who survived childhood,[2] and all his titles became extinct except the Baronies of Darcy de Knayth and Conyers, which were baronies by writ inherited from his father, and the Portuguese countship of Mértola, inherited from his mother.

David Hume wrote, "It is remarkable that this family of d'Arcy [sic] seems to be the only male descendant of any of the Conqueror's barons now remaining among the Peers.

[4] He was the only surviving son of Robert Darcy, 3rd Earl of Holderness, and his wife Lady Frederica Schomberg.