West was the second son of Mary (née Wynyard), Countess De La Warr, and Lieutenant-General John West, 2nd Earl De La Warr, Lord Chamberlain to Queen Charlotte.

He also had two sisters, Lady Georgiana West (the wife of Edward Pery Buckley and mother of Edward Pery Buckley, MP for Salisbury), and Lady Matilda West (the wife of Gen. Henry Wynyard, Commander-in-Chief, Scotland).

In 1778 he was appointed an Equerry to the Queen Consort, a post he held until 1783, when he also succeeded his elder brother William in the earldom.

On 22 April 1783, Lord De La Warr was married to Catherine Lyell (b. c. 1756; d. 27 May 1826), daughter of Henry Lyell, of Bourn, Cambridgeshire, a Swedish nobleman who had emigrated to England, and the former Catharine Allestrie (only child of George Allestrie of Alvaston, Devon).

Through his only son George (who in 1843 took by Royal Licence the name Sackville before that of West so his wife could inherit part of her father's estate), he was posthumously a grandfather of ten, nine of whom lived into maturity, including: George Sackville, Viscount Cantelupe (an MP for Helston and Lewes who died unmarried), Charles Sackville-West, 6th Earl De La Warr, Reginald Sackville, 7th Earl De La Warr, Lady Elizabeth Sackville-West (the wife of Francis Russell, 9th Duke of Bedford), Mortimer Sackville-West, 1st Baron Sackville, Lady Mary Catherine (wife of James Gascoyne-Cecil, 2nd Marquess of Salisbury and, secondly, Edward Stanley, 15th Earl of Derby), Lionel Sackville-West, 2nd Baron Sackville, William Sackville-West (who married Georgina Dodwell and had issue, including Lionel Sackville-West, 3rd Baron Sackville), and Lady Arabella Diana Sackville-West (wife of Sir Alexander Bannerman, 9th Baronet).