James was the fourth son of Loammi Baldwin Sr., and received his early education in the schools of his native Woburn and in the academies at Billerica and Westford.
He was later established there as a merchant; but the influence of his early association with the engineering faculties of the older members of his own family turned his attention in that direction.
Others high in authority differed from his conclusion, but still he was immovable in adherence to his recommendation, in spite of rejection by popular vote, to which it had been submitted, and it was not renewed till 1844, when he was again in a position of influence on the commission.
1848, he had the pleasure of seeing his plan, so long resisted, finally triumphant, and the public fountain playing for the first time in the presence of a large concourse of people.
The Boston Daily Advertiser wrote, in Mr. Baldwin's obituary, "He was of a kindly and benevolent disposition, affable in his manners, warm and unfaltering in his attachment to his friends.
He was a prominent member of the American Academy of Arts and Sciences, and during many years held the position in that learned society in the section of Technology and Civil Engineering."