His paternal grandfather, Frederick Fisher, had migrated to western North Carolina from Shenandoah County, Virginia, and had served as an officer in the local militia during the American Revolutionary War.

Although the Fishers were Lutheran or Episcopalian, their orphaned eldest daughter Christine was raised by a Catholic aunt and converted to Catholicism and later become an author of many novels, some supporting the Lost Cause.

Slow construction progress and high costs produced considerable criticism, particularly from Jonathan Worth (businessman, lawyer, Whig legislator and president of the competing Fayetteville and Western Plank Road Company).

[5] Fisher became friends with Vermont-born Sewall Lawrence Fremont (née Fish, 1816–1886), a former U.S. Army artilleryman who had become chief engineer and superintendent of the Wilmington and Weldon Railroad in eastern North Carolina in 1854.

Fremont came to command North Carolina's coastal defense and named Fort Fisher at the mouth of the Cape Fear River for his gallantry.