The primary source for Ball's life is his autobiography, Slavery in the United States: A Narrative of the Life and Adventures of Charles Ball, a Black Man, Who Lived Forty Years in Maryland, South Carolina and Georgia, as a Slave Under Various Masters, and was One Year in the Navy with Commodore Barney, During the Late War, published in 1837 with the help of Isaac Fisher.

According to Ball's autobiography, his grandfather was a man from a noble African family who was enslaved and brought to Calvert County, Maryland around 1730.

[4]: 165 The precepts of that religion are contained in a book, a copy of which is kept in each house, implying that the grandfather's African society had a high degree of literacy.

When his eldest son was four years old, he was sold to a South Carolinian cotton planter, thus separated from his wife and children, who had to remain with their legal owner in Maryland.

Shortly afterward, after the sudden death of the new husband, the new plantation, Ball, and the other enslaved people were rented out to yet another slave-master, with whom he built up a relationship of mutual trust.

Traveling by night to avoid the patrols, using the stars and his excellent memory for orientation, suffering terribly from hunger and cold, and not daring to speak to anybody, he returned to his wife and children in early 1810.

In 1813, Ball had enlisted in Commodore Joshua Barney's Chesapeake Bay Flotilla and fought at the Battle of Bladensburg on August 24, 1814.

He also related his observations of the life of his fellow enslaved people, e.g., "one very old man, quite crooked with years and labour",[4]: 88  being compelled to work although he was no longer able to keep up with the other ones who "had no clothes on him except the remains of an old shirt, which hung in tatters from his neck and arms".

On one occasion, he cites a fellow slave relating the discussion of the slaveholders on how "the greatest degree of pain could be inflicted on me, with the least danger of rendering me unable to work".

[4]: 116 While relating the first time he was driven to the Deep South, Ball frequently compares his observations there with the customs of his native Maryland.

He summarizes his observations: "The general features of slavery are the same everywhere; but the utmost rigour of the system is only to be met with on the cotton plantations of Carolina and Georgia, or in the rice fields which skirt the deep swamps and morasses of the southern rivers.

"[4]: 56  The day of his arrival on the plantation in South Carolina, he sees all his "future life, one long, waste, barren desert, of cheerless, hopeless, lifeless slavery; to be varied only by the pangs of hunger and the stings of the lash".