James Ladson was born in 1753 in Charleston to a prominent South Carolinian family of English origin.

His great-grandfather John Ladson emigrated from Northamptonshire in England to Barbados and then to Carolina as one of the first English settlers in 1679, where he built a large plantation and served in the Commons House of Assembly from 1685.

[2] Following the early death of his parents he was raised by his uncle John Gibbes (1733–1780), who owned the Grove Plantation that included today's Hampton Park and its surrounding neighbourhoods.

[2][3] He served as an officer during the American Revolutionary War from 1775 to 1780, first alongside his childhood friend Thomas Pinckney and eventually as a captain in the 1st regiment of the Continental Line.

[10] Art historian Maurie McInnis notes that "she visually made reference to the taste of the slave women around whom she had been raised" with the turban and bright colours.

[8] James Ladson was the father of James H. Ladson (1795–1868), a major plantation owner who by 1850 owned over 200 slaves who produced 600,000 pounds of rice each year on his La Grange and Fawn Hill plantations,[11][12] who was also the Danish Consul in South Carolina.

Among their descendants are Ursula von der Leyen, who for a year lived under the name Rose Ladson in London to escape terrorists.