A Unitarian minister, he played a leading role in the early history of Harvard Divinity School, and he later became involved in politics as a State Representative and U.S.

Palfrey began school at the Berry Street Academy in Boston and studied Greek and Latin with William Ellery Channing.

In 1831, after Andrews Norton retired in 1830, he became a Professor of Biblical Literature and Dean of Faculty at the Divinity School and removed to Cambridge.

In this position, in addition to teaching the Bible, Hebrew, and other semitic languages, he was in charge of the building, organized faculty meetings, and was the chief disciplinarian.

He made changes, including instituting a new set of rules for the school, reorganizing the curriculum, and dealing with complaints from faculty members about their salaries.

As a professor, he taught Unitarian ministers such as Andrew Peabody, William Henry Channing, James Freeman Clarke, Chandler Robbins, William Greenleaf Eliot, Cyrus A. Bartol, Charles Timothy Brooks, George Edward Ellis, Abiel Abbot Livermore, Theodore Parker, Henry Whitney Bellows, Edmund Hamilton Sears, Rufus Phineas Stebbins, the philosopher William Dexter Wilson, the artist Christopher Pearse Cranch, the music critic John Sullivan Dwight, and the Swedenborgian Benjamin Fiske Barrett.

He was a "Conscience Whig" who opposed slavery, having freed sixteen slaves inherited from his father, who, like his two brothers, was a successful Louisiana plantation owner.

His anti-slavery views alienated him from more conservative members of his district, such as the "Cotton Whigs," and in 1848 he was unsuccessful in his campaign for re-election on the Free-Soil ticket.

In addition to opposing slavery, Congressman Palfrey advocated for the rights of free blacks traveling in the South and attempted unsuccessfully to remove provisions that limited suffrage to whites in Oregon's Territorial Constitution.

After Abraham Lincoln's election in 1861, Senator Charles Sumner secured him appointment as Postmaster of Boston, a post he held from 1861 to 1867.