Released before the 2003 season, he signed as a free agent with the Milwaukee Brewers, where he became the closer, converting 60 of 67 save opportunities in 2003 and 2004 and making the 2004 NL All-Star team.

Before the 2005 season, the Atlanta Braves returned closer John Smoltz to his original starter role to compensate for several losses in their pitching rotation.

However, Kolb endured a poor season with the Braves, going 3-8 with a 5.93 ERA while recording only 11 saves before being replaced in the closer role by Chris Reitsma.

At the 2005 Major League Baseball Winter Meetings, Kolb was traded back to Milwaukee in exchange for reliever Wes Obermueller.

Kolb signed a minor league contract with an invitation to Spring Training with the Boston Red Sox on January 22, 2008.