Among his other ancestors were governors John Yeamans, James Moore, Robert Gibbes, Thomas Smith and Joseph Blake, and the first European settler of Carolina Henry Woodward.
"[9] The Charleston Daily News described him as "an excellent specimen of the old Carolina gentleman, pure in character, and high-toned in his dealings, and was for many years the head of the house of James H. Ladson & Co., now represented by the firm of W.C. Bee & Co.
He was also a bank director, and for most of his life a leading member of St. Michael's Episcopal Church of our city, where he stood foremost in Christian virtues and active benevolence.
"[10] James H. Ladson was part of the oligarchy of elite planters and merchants who controlled Charleston, a city that played a major role in the history of slavery in the United States, and he and several of his family members were actively involved in launching the American Civil War.
[14][15] His sister Sarah Reeve Ladson was a prominent socialite who was married to the Baltimore merchant, ship owner, East India importer and art collector Robert Gilmor, Jr.[16] Sarah Reeve Ladson was regarded as one of the most fashionable American women of her time, and was the subject of several portraits and sculptures, including a famous portrait by Thomas Sully.
[17] Art historian Maurie D. McInnis notes that "she visually made reference to the taste of the slave women around whom she had been raised" with the turban and bright colours.
[12][23][24] The religious and moral instruction of the negroes has, for several years, been a subject of great interest to me, and I am satisfied that our exertions in their behalf (although much, very much, yet remains to be done) are not only misunderstood abroad, but unappreciated.
They are naturally dull, and of a weak intellect, but generally possessing good memories; and those who have been engaged in this work of charity, have to lament, after much labour, that the instruction they have endeavoured to give, although remembered, has been perverted and misdirected.Ladson believed strongly in religious instruction to maintain discipline among the slaves and built his own chapel on the plantation that could accommodate around 100–110 slaves at a time.