As a young girl in 1672, his great-great-grandmother Ann Smith was an accuser of Goody Cole, the only woman in New Hampshire convicted of witchcraft.

Clifford was opposed to a high tariff, supported internal improvements, endorsed state banking, and was in favor of federal retrenchment.

In 1846, President James K. Polk appointed him 19th Attorney General of the United States after his predecessor, John Y. Mason, returned to being Naval Secretary.

On December 9, 1857, President James Buchanan nominated Clifford as an associate justice of the United States Supreme Court, to a seat vacated by Benjamin R.

As a longtime partisan Democrat, the opposition labeled Clifford a political hack and a "doughface" — a Northern man with Southern sympathies.

[8] Anti-slavery representatives in the United States Senate fiercely opposed Clifford due to his pro-slavery record.

Justice Clifford rarely declared any legal philosophy about the U.S. Constitution but believed in a sharp dividing line between federal and state authority.

Senator James W. Bradbury, said Clifford's view was that the Constitution was not an "elastic instrument to be enlarged or impaired by construction, but to be fairly interpreted according to its terms, and sacredly maintained in all its provisions and limitations, as the best guaranty for the perpetuity of our republican institutions.

[citation needed] Clifford's fields of expertise were commercial and maritime law, Mexican land grants, and procedure and practice.

During Reconstruction, Clifford continued his skepticism of the federal government, now unrestrained by any consideration for the exigencies of wartime emergency powers.

[14] Clifford joined the majority in a 5–3 decision holding that the Legal Tender Act could not constitutionally apply to preexisting debts.

The lower court ruled that the plaintiff must be repaid in paper money and that the defendant had to pay the difference in the valuation of the goods in gold to greenbacks.

[17] Justice Clifford, joined by Justices Field and Nelson, dissented from the grant of certiorari, stating publicly, "I dissent from the order of the Court in these cases, especially from that part of it which opens for re-argument the question whether... the Legal Tender Act is constitutional as to contracts made before its passage—as I hold that the question is conclusively settled by the case of Hepburn vs Griswold..."[18] On May 1, 1871, the Court ruled 5–4 to overturn Hepburn v. Griswold and find the Legal Tender Act constitutional — facially and as applied to pre-existing debt.

In Hall v. DeCuir (1878), Justice Clifford wrote a separate concurrence to uphold segregation on steamships, coining the phrase "equality is not identity."

Clifford believed that the commission erred in nullifying Tilden's apparent victory and never accepted Hayes as the lawful president.

In this instance Clifford put the country before his strong party beliefs, and his personal hope of having a Democratic president choose his successor.

Justice Samuel Miller wrote that Clifford's mental deterioration was "obvious to all of the Court" and "in the work we do, no man ought to be there after 70."