In the winter of 2008, local media reported that he was a target for English side Derby County, but problems with the work permit stopped the transfer.

[3] After many departures from the club in the 2008 summer transfer window, Ivanov became an integral part of CSKA's defence, together with Kiril Kotev.

[6] He made his competitive debut for the club on 13 July 2011, in a 4–0 home win against Shkëndija in the second qualifying round of the Champions League.

On 30 August 2012, Ivanov scored a header against Tromsø to give Partizan a 1–0 win and thus qualify the team for the Europa League group stage.

Ivanov played his domestic league debut for the club in the home game in the St. Jakob-Park on 11 August as Basel were defeated 2–1 by Zürich.

[10] On 11 December 2013, in the Champions League group stage away match Ivanov was shown the red card and dismissed during Basel's 2–0 defeat by Schalke 04.

[11][12] On 14 January 2014, Ivanov suffered a knee injury in a friendly match against Eintracht Braunschweig,[13] causing him to miss the rest of the 2013–14 season.

To the start of the 2014–15 season he was still not able to play football due to the injury reoccurring and on 16 September the club announced that Ivanov had to undergo a second suggery (anterior cruciate ligament reconstruction) but the operation had been successful.

[15] Ivanov had expected to recover for the start of the Euro 2016 qualifiers,[16] receiving a call-up for the match against Azerbaijan, but was eventually deemed not fully fit.

[21][22] However, after Marinos Ouzounidis took over, he was no longer a first-team choice for the club, especially after the defensive mistake that helped Apollon Smyrnis open the scoring in the Greek Cup match between the two[23] (Panathinaikos eventually won 4–3); instead, discussions within Panathinaikos were made as to whether Ivanov would remain at the club even as a backup for the first team or would be completely released from his contract.

[32] Ivanov made his debut for the senior national team on 20 August 2008, in a 2–1 win against Bosnia and Herzegovina, during which he entered the match as a substitute in the 67th minute.

On 10 October 2009, Ivanov earned his fourth cap in a 4–1 away loss against Cyprus, attracting some criticism from the media for defensive mistakes.

After missing the qualifiers for Euro 2016 through injury, Ivanov made his return to the team on 6 September 2016, in the 4–3 win over Luxembourg in a 2018 World Cup qualification match.