[2][3] A member of the prominent Ladson family, her father was a military officer in the American Revolutionary War and served as the Lieutenant-Governor of South Carolina.

Her mother, Judith, was a daughter of Benjamin Smith, a South Carolina slave trader, planter, banker and speaker in the colony's Royal Assembly.

Through her mother, Ladson was a descendant of Thomas Smith, a colonial governor of South Carolina, and Joseph Wragg, a slave trader and politician.

[16] Maurie D. McInnis, an art historian, noted that Ladson "visually made reference to the taste of the slave women around whom she had been raised" with the turban and bright colours portrayed in Sully's portrait of her.

[6] Sully's portrait of Ladson has been exhibited in Grandeur Preserved: Masterworks Presented by Historic Charleston Foundation in New York, and Art in America: Three Hundred Years of Innovation in Shanghai and Beijing.