After his family moved to Philadelphia, he rebelled against them, gambled and drank, and aged 20 went to work at the Huntington shipyards in Newport News, Virginia.
He spent time observing at the local court, where judges gave summary justice to African-Americans, generally poorly-educated, who were charged with various misdemeanours and felonies.
After serving in the Spanish-American War, Kelly returned to Newport News and opened a cafe and gambling den.
[1][2] He developed a talent for storytelling, drawing on the mannerisms and speech of the judges he had witnessed, and made his performing debut in New York City in 1900.
[1] Although Kelly did a number of different Southern dialects playing the various characters in his "Virginia Judge" sketches, an Irish flavor sometimes sneaked through.