He followed Dungy to Indianapolis in 2002 and remained with him for his entire tenure, helping lead the Colts to a win in Super Bowl XLI.

On January 24, 2010, Caldwell became the fifth rookie head coach to lead his team to the Super Bowl after securing a 30–17 win against the New York Jets in the AFC Championship Game.

They lost their first game in the playoffs to the New York Jets, 17–16, on January 8, 2011, marking the end of the Peyton Manning era in Indianapolis.

Starting quarterback Peyton Manning missed the entire season due to undergoing neck surgeries and without him the Colts appeared to be a rudderless team.

Thirteen days after his dismissal from the Indianapolis Colts, Caldwell was named quarterbacks coach by the Baltimore Ravens on January 30, 2012.

[6] On the day after defeating the New England Patriots in the AFC championship game, head coach John Harbaugh announced on January 22, 2013, that Caldwell would be the team's permanent offensive coordinator going into the 2013 season.

[7] On February 3, 2013, Caldwell helped lead the Ravens to a 34–31 victory over the San Francisco 49ers in Super Bowl XLVII at the Mercedes-Benz Superdome.

Halfway through the 2015 season, the Lions had struggled to a 1–7 record, and both team president Tom Lewand and general manager Martin Mayhew were fired.

[12] There was speculation that Caldwell would soon be fired as well,[13] but the next week the Lions won a road game against the Green Bay Packers for the first time since 1991.

In 2016, Caldwell's third season in Detroit, the Lions improved to a 9–7 record and lost to the Packers in a Week 17 game that determined the winner of NFC North.

After being released from Detroit at the end of the 2017 NFL season, Caldwell was hired by the reincarnated XFL to a consulting panel that addressed football rules for the league.

[19] Caldwell was interviewed in December 2018 by the Green Bay Packers for their open head coach position that was previously held by Mike McCarthy until being let go mid-season in 2018.

[26] Caldwell not receiving another head coaching opportunity since 2017, along with his dismissals from the Colts and Lions, were cited by Flores as examples of the league's alleged racial discrimination in his 2022 class-action lawsuit against the NFL.