[7] Shortly after taking his seat in the House of Lords he voted for Catholic emancipation, which brought him into conflict with many of the clergy in his diocese.

In 1851, Sumner led the religious service at the formal opening of the Great Exhibition in "The Crystal Palace" in Hyde Park.

[8] In the Gorham Case, Sumner came into conflict with Henry Phillpotts, Bishop of Exeter (1778–1869), who accused him of supporting heresy and refused to communicate with him.

[8] His obituary in the Norfolk News of 13 September 1862[9] commented that "he strongly opposed the admission of Jews into parliament ... and was among the foremost to denounce the Puseyite school of theology".

[11] On 31 March 1803, Sumner married Marianne Robertson (1779-1829) in the parochial chapel of St Mary Walcot, Bath, Somerset.

A recumbent effigy by H. Weekes, R.A., funded by public subscription after Sumner's death, is in the nave of Canterbury Cathedral.