He succeeded to the baronetcy after his eldest brother the 2nd Baronet died without issue in 1826.
On 26 February 1826 he married Hannah Jean Grey, third daughter of Sir George Grey, 1st Baronet, of Falloden, KCB, Commissioner at Portsmouth Dockyard, and Mary Whitbread, daughter of Samuel Whitbread.
One of their twelve children was Admiral Francis George Kirby (1854–1951), Captain of HMS Phaeton.
Sir Henry was, during his life, Curate in charge at Holy Trinity Church, Bembridge, Isle of Wight; Rector of the Church of Holy Trinity, Fareham, Hampshire (the building of which had been paid for by himself and his mother, Lady Jane Thompson), and in 1845 he was given the living of Frant, Sussex, by the Earl of Abergavenny.
He died in Chichester in 1868 and was buried at Holy Trinity Church, Fareham.