A member of the Republican Party, he was a State Senator in New York from 1862 to 1869 and served as the 34th U.S. Secretary of the Treasury from November 14, 1881 until his death in 1884.

After the death of Sanford E. Church, Folger was appointed Chief Judge by Governor Alonzo B. Cornell on May 20, 1880, to fill the vacancy temporarily.

In 1881, President James Garfield offered Folger the position of United States Attorney General, which he declined.

Later that year, he resigned from the bench to accept an appointment by President Chester Arthur to serve as Secretary of the Treasury.

In 1883 he appointed Mifflin E. Bell to the Office of the Supervising Architect for the U.S. Treasury, an agency that designed federal government buildings from 1852 to 1939.