Jolly made his major league debut in relief on May 9, 1953, against the Chicago Cubs at Milwaukee County Stadium.

From 1953 to 1957, the first five years that the Braves were in Milwaukee, he was second on the pitching staff with 158 relief appearances, an average of almost 32 per season.

During those seasons the closer's job was held at different times by Lew Burdette, Ernie Johnson, Jolly, and Don McMahon.

On October 15, 1957, Jolly was purchased from the Braves by the San Francisco Giants, but never again pitched in a big league game.

Jolly died in 1963 at the age of 38 in Durham, North Carolina, one year after he underwent surgery for a brain tumor.