Born in Springfield, New Brunswick, James was the eldest of the six children of Francis Daniel Ganong and Deborah Ruth Keirstead.

His father was a farmer and a merchant and as a young man, James worked as a shopkeeper and as a travelling salesman.

In 1873, he and his brother Gilbert moved to the border town of St. Stephen on the St. Croix River across from Calais, Maine.

There, they established a grocery business and within a few years added a bakery and confectionery manufactory plus expanded their retailing to include a store in Calais.

Gilbert Ganong retained the store, bakery and confectionery business while James took over the soap factory.