[2] Vice-Chancellor of Cambridge University in 1814,[3] In 1816, Kaye was elected Regius Professor of Divinity and he revived public lectures on theology, with an focus on the study of ecclesiastical history and the Early Church Fathers.

His first series of lectures, The Ecclesiastical History of the Second and Third Centuries, illustrated from the Writings of Tertullian was published at Cambridge in 1825, and was followed by published lectures on Justin Martyr (1829), Clement of Alexandria (1835) and The Council of Nicæa in connection with the Life of Athanasius (posthumously, 1853).

[citation needed] Kaye was appointed Bishop of Bristol in 1820, and remained there until his translation in 1827 to Lincoln.

He was opposed to the revival of the Convocations of Canterbury and York and was sympathetic to George Cornelius Gorham's views on baptism.

[4][5] Within Lincoln Cathedral, Kaye is commemorated by a recumbent effigy tomb monument designed by Richard Westmacott.