Foster was born outside of Tiffin, Ohio, and grew up in the western Seneca County boomtown of Rome.

Foster was unsuccessful in a bid to return to the House in 1890, but was appointed by Benjamin Harrison a year later to become Secretary of the Treasury upon the death of William Windom.

During the transition, after Grover Cleveland won the presidential election, Harrison intimidated to the United States Congress that there was a pending economic crisis.

[2] One of the first clear signs of financial crisis came on February 20, 1893, twelve days prior to Cleveland's inauguration, when receivers were appointed for the Philadelphia and Reading Railroad, which had greatly overextended itself.

Instead of addressing the economic crisis, Foster spent his final days as secretary posing for his official portrait.