Drury was educated at Westminster School and Trinity College, Cambridge.
[1] Admitted to Trinity in 1768, he was unable to continue at Cambridge due to lack of means, and in 1769 became an assistant master at Harrow School.
[2] In 1771, Drury turned down the chance to join Samuel Parr's breakaway school at Stanmore.
[1] Drury succeeded Benjamin Heath the younger, his brother-in-law, as headmaster of Harrow School in 1785.
Of their three sons Henry Joseph Thomas Drury (1778–1841) was a master at Harrow, and Benjamin Heath Drury (1782–1835), became an assistant-master at Eton College; and their daughter Louisa Heath Drury married John Herman Merivale.