He was the son of William Robert Spencer and Countess Susan von Jenison-Walworth.

His maternal grandparents were the former Charlotte Smith and Count Francis von Jenison-Walworth, Chamberlain to the Elector Palatine.

His maternal uncle, Count Franz von Jenison-Walworth, married Mary Beauclerk, a daughter of Topham Beauclerk (a great-grandson of King Charles II) and the former Lady Diana Spencer (his maternal grandfather's sister, both children of the 3rd Duke of Marlborough), and served as Chamberlain to the King of Württemberg.

He became Perpetual curate of Buxton until 1829, when he became Rector of Leaden Roding until[4] his appointment as the Bishop of Madras in 1837.

[5] Resigning his See in 1849, he was afterwards minister of the Marbœuf Chapel (English Protestant) in Paris; an assistant Bishop of Bath and Wells (4 October 1852 – 10 May 1853) and commissary during Bagot's illness;[6] and then Chancellor of St Paul's Cathedral[7] from 1860.