In his second season with Stabæk, the then-Norway U21 international attracted interest from English clubs Leeds United and Wimbledon.

A transfer to Wimbledon seemed likely, but it fell through after Olsen was tackled by Heiðar Helguson in a league match and sustained a broken tibia and fibula.

[1] In Stabæk's return season to the Tippeliga in 2006, Olsen was one of the most important player for his team, and was linked to one of the favourites to win the league, Brann from Bergen.

In November 2011, Olsen along with Truls Haakonsen were charged as part of a police investigation into the Veigar Páll Gunnarsson transfer 2011,[3] but acquitted in court.

In December 2019 it was announced that the upcoming transfer window was his last as chief of Stabæk's player logistics, and that he would become director of sports at Danish club AaB in April 2020.