Born on October 13, 1817, in Milford, Sussex County, Delaware,[1][2][3] Fisher attended the public schools of Kent County, Delaware, Mount St. Mary's College (now Mount St. Mary's University) in Emmitsburg, Maryland,[4] then graduated from Dickinson College in Carlisle, Pennsylvania, in July 1838.

[3] He read law with John M. Clayton, then the Chief Justice of the Delaware Supreme Court,[1] and was admitted to the bar in 1841.

[3] In Congress, Fisher supported Abraham Lincoln's compensated emancipation proposal, but failed to find someone in the Delaware General Assembly willing to introduce it.

"[2] However, he was appointed by President Benjamin Harrison on May 31, 1889, to serve as first auditor for the United States Department of the Treasury until March 23, 1893.

[3] Fisher "then returned to the home of his childhood, lived quietly in his extensive library, and devoted the last years of his life to reading and literary pursuits.