[2] Torrey was a great-grandson of Manasseh Cutler who had been a member of the United States Congress.
He moved to the household of his uncle Joseph Torrey to go to Burlington High School.
He then studied at the University of Vermont, completing his bachelor's degree in 1858.
After teaching school for a time in Beverly, Massachusetts, he went to the Union Theological Seminary in the City of New York from which he graduated in 1864.
A "sound philosopher",[3] Torrey was an Intuitionist, as well as possessing a strong interest in Immanuel Kant, having "cut his philosophical teeth" on the Critique of Pure Reason.