The first courthouse in Carrollton was burned on April 5, 1865, by troops of Union General John T. Croxton.

Incorporated in 1831, the town was named after Charles Carroll of Carrollton, Maryland, the only Roman Catholic and longest-living signer of the Declaration of Independence.

It was part of a program of intimidation and racial terrorism, with these murders frequent in the decades on either side of the turn of the 20th century.

Among the numerous African Americans lynched in Carrollton was John Gibson, hanged on August 28, 1907.

As of the 2020 United States census, there were 1,023 people, 353 households, and 218 families residing in the town.

The racial makeup of the town was 53.39% White, 44.58% Black or African American, 0.10% Asian, and 1.93% from two or more races.

Carrollton is also the home of the Pickens County Educational Center, a branch of Bevill State Community College.