After spending his early years in farm work, and acquiring a knowledge of mathematics in his spare time, Phillips became a master at the grammar school in Woodbridge, Suffolk, and later at that in Worcester.

In 1824 Phillips left Worcester in order to enter Magdalen Hall, Oxford, but after a short residence migrated to Queens' College, Cambridge, in October 1825.

On the death of Joshua King in 1852, he was elected president of Queens' where he lived for the rest of his life.

As a fellow of Queens', Phillips had at first continued to study mathematics, but soon turned to Hebrew, which he began to teach in the college, although there was then little interest in the subject at Cambridge.

In August 1848 he married Emily Frances (died 1898), daughter of Henry Pilkington, of Tore, County Westmeath, Ireland; they had no children.