[4] He began his career playing for Malmö FF youth teams before moving to the academy of BK Olympic.

[5] He made his Allsvenskan debut on 1 October 2012 in an away fixture against Syrianska FC, when first choice goalkeeper Johan Dahlin was suspended, and kept a clean sheet as the match ended 2–0 in Malmö's favour.

Copenhagen wanted to make the loan move permanent,[15] and on 24 May 2016 Olsen signed a four-year contract with the club for a transfer fee of almost €600,000.

[16] In July 2018, Olsen signed a five-year contract with the Italian club A.S. Roma with a reported fee worth up to €12,000,000,[17] replacing the outgoing Alisson Becker.

[18] Olsen was eventually replaced by veteran Antonio Mirante by interim manager Claudio Ranieri towards the end of the season.

In total, Olsen made 35 appearances in his sole season at Roma, conceding 58 goals and keeping just seven clean sheets.

Despite Carlo Ancelotti's initial desire to bring the 31-year-old to Goodison Park on a permanent basis, the surprise exit of the Italian in the summer seemingly scuppered any hope of that taking place.

While Everton instead brought both Asmir Begovic and Andy Lonergan to Merseyside to provide back-up to Pickford, Olsen opted to move to Sheffield United on loan.

[31] Manchester City released a statement saying that "the club has launched an immediate investigation and once identified, the individual responsible will be issued with an indefinite stadium ban.

[34] On 29 October 2022, he made his first appearance since signing for Villa permanently in an away defeat to Newcastle United, coming off the bench to replace Emiliano Martínez, who had suffered a suspected concussion.

[39] In March 2021, Olsen and his family were threatened with a machete by a masked gang during a raid on their home in Altrincham, Greater Manchester.