He was the oldest surviving son of William Villiers, 3rd Earl of Jersey, and his wife, the former Lady Anne Egerton, the daughter of Scroop Egerton, 1st Duke of Bridgewater, and widow of Wriothesley Russell, 3rd Duke of Bedford.

He was a Lord of the Admiralty from 1761 to 1763 and was sworn of the Privy Council on 11 July 1765 and served as Vice-Chamberlain from 1765 to 1769.

On his elevation to the peerage in 1769, he was made a Gentleman of the Bedchamber to George III (1769–1777) and served as Master of the Buckhounds (1782–1783) and in other court posts until 1800.

Lord Jersey married Frances Twysden at her stepfather's house in the parish of St Martin-in-the-Fields on 26 March 1770.

Lady Jersey, who was seventeen years younger than her husband, became one of the more notorious mistresses of George IV in 1793, when he was still Prince of Wales.