Smith played college football for the Wake Forest Demon Deacons, and received consensus All-American honors.

As a redshirt freshman in 2005, Smith started all 11 games and finished the season with 60 tackles (52 unassisted), three interceptions, two sacks, 12 pass breakups and one forced fumble.

In 2006, the sophomore played 14 games with three starts, registering 48 tackles (37 solo), three interceptions, four sacks, 11 pass breakups and one forced fumble.

He tied for first in the nation with a school-record eight interceptions and was first-team All-Atlantic Coast Conference (ACC) selection and received second-team All-America honors from Walter Camp.

The following week, he beat Green Bay Packers receiver Greg Jennings on a jump ball for his second career interception.

In a home game against the St. Louis Rams on October 10, Smith returned an interception 42 yards for a touchdown in the Lions' 44–6 win.

While the Lions were leading in the third quarter, Smith taunted New England quarterback Tom Brady with a finger wag after an incompletion.

Brady would target Smith on three of his next four touchdown passes, as the Patriots ultimately scored 24 points unanswered to win 45-24.