He owned a large shipping line that ran between London and Philadelphia, but was exiled and imprisoned during the Revolutionary War because of his Quaker beliefs.
Fisher's father also purchased a country estate north of the city overlooking the Schuylkill River from the east and built a house there in 1753 called "The Cliffs."
Customers were able to order items such as porcelain, silverware, brass pulls for dressers, and every other imaginable type of merchandise from a detailed catalog.
He traveled widely in America and England (1767-68) and made notes on the manufacture of textile, glassware, and ceramic items for inclusion in the catalog.
However, Fisher, who by then had begun a course towards eccentricity, continued to show opposition to the revolutionary cause, and was arrested in 1779 on the charge of being a Tory on the basis of a letter he sent to his brother Jabez Maud in New York City.
Many Quakers, and even some of his family, opposed his strong stand against authority and the revolutionary fervor, and at one point he was threatened to be "read out of Meeting" (disowned).
He traveled to Bristol, England, Nottingham, Manchester, Sheffield, Warrington, and elsewhere to visit textile mills and other manufacturers, comparing their quality and prices.
On May 20, 1793, Fisher traveled by stagecoach to New York, then took a boat to Newport, proposing to Hannah immediately upon arrival at the Rodman house.
The Fishers spent summers at The Cliffs which, perched high above the Schuylkill River, was cooler and more free of mosquitoes than their city dwelling, and the Wharton estate was also nearby.