Born in County Carlow in Ireland, Smith's father died when James was young.
During the U.S. Civil War, he served for a year as a musician with the 2nd Maryland Infantry Regiment, before returning to tailoring.
[1] In 1864, Smith moved to Springfield, Illinois, where he later joined the Journeymen Tailors' National Union of the United States.
In 1885, he was also elected as a vice-president of the Federation of Organized Trades and Labor Unions (FOTLU).
[2] In 1886, FOTLU was reorganized as the American Federation of Labor, and Smith was elected as one of its vice-presidents.