Playing in the Brøndby reserve team, he made his senior debut for Brøndby IF in August 2006, replacing Morten "Duncan" Rasmussen and scoring two goals in the 4–0 victory over FC Flora Tallinn in the second qualification round of the UEFA Cup, after being subbed on in the beginning of the second half.

After his return from a loan spell at SønderjyskE, Agger became a regular part of the Brøndby squad and played 26 league games in the 2012–13 season.

On 31 August 2011, Agger joined Swedish club Djurgårdens IF on loan to the end of the season with a buying obligation.

On 29 June 2014, Vejle Boldklub announced that Agger had been sold to newly promoted Silkeborg IF a year before the end of his contract.

[10] In October 2016, Agger suffered an anterior cruciate ligament injury to his left knee, which would rule him out for six months.

[13] He reached promotion to the Danish 1st Division in his first season with Hvidovre, scoring 19 goals in 27 league appearances.