Joseph's paternal grandfather, William Harris (1757 – 1812), had been a Continental Army officer in the American Revolutionary War and thereafter, as well as a member of the Pennsylvania General Assembly.

Looking to prolong his life and also leave his wife with a means of supporting herself, in 1850 Stephen Harris sold his farm and moved his family to Philadelphia, Pennsylvania.

[8] In 1853, even before graduating from Central High School, Harris took a job as a topographer for the Easton and Water Gap Railroad (which became the North Pennsylvania Rail Road Company later that year), then under construction.

[2]: 3  He left this job after a year, becoming an astronomer for the United States Coast Survey, whose superintendent, Alexander Dallas Bache, had been president of Central High School.

Although he displayed many quirks of personality, Joseph Harris was meticulous in his work; his autobiography provides, among other things, an idea of U.S. Coast Survey shipboard life in the 1850s.

[9] The trip south was not without its hardships: Harris suffered from diarrhea on the Mississippi River and within a few days of his arrival at New Orleans, Louisiana, he contracted typhoid fever, which nearly killed him.

The work of the surveyors was made difficult by the large populations of insects—everything from mosquitoes to flying cockroaches—that inhabited the coastal swamps and marshes, by the dearth of clean water, by the arrest of some of the crew after a brawl, and by hurricanes, all of which are described in Harris's autobiography.

[citation needed] During his year on the Phoenix, Harris and his crew performed triangulation along the coast from Pascagoula, Mississippi, to the entrance to Lake Pontchartrain, a distance of about sixty miles (97 km).

[10] Harris took a similar position with the Kentucky Geological Survey, but he resigned after one month in July 1856 and returned to the Gulf of Mexico to complete his earlier work.

The American survey team sailed from New York City on April 20, 1857, and proceeded to Panama (then a part of Colombia) by way of Kingston, Jamaica.

At Panama City, they boarded the vessel John L. Stephens and sailed for San Francisco, California, with several stops in Mexico along the way before arriving on May 15, 1857.

Because of the damage Uncas sustained during the gale, Harris was ordered to transfer his equipment and crew to her sister ship, USS Sachem, for the remainder of the voyage to the Gulf Coast.

There, Harris was rebuffed by the Navy supply department and was instead ordered, under threat of facing a firing squad, to support an expedition to Edisto Island.

Harris declined, repeatedly stating that he was under Coast Survey orders to proceed to Ship Island, Mississippi, and report to Commodore David Farragut.

Only through the personal intervention of Commodore Samuel Francis Du Pont was Sachem finally coaled and allowed to depart Port Royal.

In April 1862, Harris and the other surveyors marked navigable channels in the river and established survey markers on the shore to serve as control points for indirect mortar fire into the forts defending the approaches to New Orleans.

Commander Porter wrote to Alexander Dallas Bache, superintendent of the Coast Survey, concerning the battle of Forts St. Philip and Jackson: The results of our mortar practice here have exceeded anything I ever dreamed of; and for my success I am mainly indebted to the accuracy of positions marked down, under Mr. Gerdes' direction, by Mr. Harris and Mr. Oltmanns.

[18] Harris returned to railroad work around 1864, entering private practice as a civil and mining engineer[19] and also joining his older brother Stephen in the Schuylkill Company of Pottsville, Pennsylvania.

[20] This work exposed them to danger in the form of the Molly Maguires, members of an Irish-American secret society who were active in the coal fields of Pennsylvania at the time.

J. P. Morgan, who owned or controlled a considerable portion of the P&R's stock and debt, chose Harris, known to be a fiscal conservative, as one of the company's receivers, and later its president.