In March 2021 he announced to have signed with Bressoux playing in the Belgian Provincial Leagues, but early August of that same year he instead retired and became goalkeeper manager at FC Differdange 03.

After a series of good performances, Bailly was rewarded on 2 December with a one-year extension, giving him a contract until the end of the 2013–14 season.

[7][8] He made his debut on 18 July 2015 in a pre-season friendly against SD Eibar and saved a penalty kick from Dani Nieto during a 4–1 win for Celtic.

[9] He began the season as second choice to Craig Gordon, but made his competitive debut in the Scottish League on 22 August 2015 in a 3–1 win away at Dundee United.

[16][17] He received his first call up to the full international squad in the summer of 2007 for a Euro 2008 qualifying tie against Portugal, although he did not play.

[18][19] He featured in several more squad selections before winning his first cap on 10 October 2009 in a 2010 FIFA World Cup qualifying tie at home against Turkey.