Goss was born April 24, 1823, in Lancaster, New Hampshire, received a public school and academic education, and learned the printer's trade.
He worked as a printer until he moved with his family to Wisconsin Territory in 1841, and eventually purchased and farmed 160 acres of land near the east end of Pewaukee Lake.
On January 21, 1851, he was married in Pewaukee to Abby B. Bradley, a native of Cayuga County, New York; they had one child, Clara F. Goss.
After the 1855 Assembly session ended (Goss was succeeded in the next session by Democrat James Weaver of Lisbon), he moved to Freeport, Illinois, to join his brother N. S. Goss in operating a large grocery store for about one year; he then moved to Waverly, Iowa, and went into the real estate business for about two years, after which he moved on to Neosho Falls, Kansas, where he (along with his brother N. S. Goss and two others) formed a company, purchased several hundred acres of land, platted Neosho Falls, built a dam, erected mills, and made other improvements.
He was an avid birder and amateur ornithologist (as was his brother N. S. Goss); they traveled widely throughout the U.S., and gathered a collection that represented 720 species of birds.