[4][5] From 1749 to 1754 and again from 1755, Cadogan was returned on the interest of his father-in-law, Lord Montfort, as a Member of Parliament for Cambridge until he inherited his father's title in 1776.

[5] He was also appointed Keeper of the Privy Purse to Prince Edward in 1756, Surveyor of the King's Gardens from 1764 to 1769 and Master of the Mint from 1769 to 1784.

[5] In 1774, Cadogan entered into an agreement to loan £20,500 to Sir Robert Cockburn, Sir George Colebrooke, John Stewart and John Nelson to enable the four men to purchase a 320-acre estate in St George Parish, Grenada, together with the enslaved people on the estate.

[2] In 1777 he leased 100 acres (0.40 km2) of the family estate in Chelsea to architect Henry Holland for building development.

Together, they had six children:[9][unreliable source] Cadogan's first wife died in 1768, and on 10 May 1777, he married Mary Churchill, a daughter of Col. Charles Churchill and Lady Mary Walpole (a daughter of former Prime Minister Robert Walpole).