Pelham was born on Stratton Street, Piccadilly, the son of Thomas Pelham, 2nd Earl of Chichester and Lady Mary Henrietta Juliana Osborne.

He was educated at Westminster School and Trinity College, Cambridge.

[1] Pelham was commissioned a cornet in the 6th (Inniskilling) Dragoons on 24 June 1824, transferring to the Royal Horse Guards on 14 October of that year.

Lord Chichester married Lady Mary Brudenell, daughter of Robert Brudenell, 6th Earl of Cardigan, at St. Mary's Church, Cadogan Street, London, on 18 August 1828.

They had seven children: Lord Chichester died at the family estate of Stanmer Park and was succeeded by his eldest son Walter.