His father, Laurenz Fehrman, was a Swedish military officer who served under King Charles XIV John in the Coalition Wars.
When the King changed sides, he fled to Scotland, where he married James' mother, Mary Farquharson Black.
In 1842, having shown some aptitude for drawing, he began attending evening classes at the National Academy of Design.
Over the next ten years, he became increasingly involved in the Abolitionist movement and discovered that he had a talent for public speaking; attending a course on the subject taught by Edward Delafield Smith.
He was also reprimanded for insulting a superior officer, General William High Keim; calling him a "Pennsylvania Dutchman" who obtained his position through political influence.
This would involve planned events, to which he invited journalists, and populist-themed lectures; attacking the established, academic art scene.