Since the death of Baldwin's uncle Emperor Henry in 1216, the Latin Empire had declined and the Byzantine (Nicene) power advanced; and the hopes that John of Brienne might restore it were disappointed.

He went to the West in 1236, visited Rome, France and Flanders, trying to raise money and men to recover the lost territory of his realm.

In 1237, with the support of the King of France and the Countess of Flanders, he chased his sister Margaret from power to become the next Count of Namur.

In around March 1238, Baldwin II's regency council pawned the Crown of Thorns to the Venetian Podestà of Constantinople for 13,134 hyperpyra from a "consortium of creditors".

The extremity of his financial straits reduced him soon afterwards to handing over his only son, Philip, to Venetian merchants as a pledge for loans of money.

On the night of 24 July 1261, a group of soldiers under Alexios Strategopoulos entered Constantinople through a secret passageway and captured the city.

Baldwin was asleep in the Blachernae Palace when the noise of the fighting awoke him; upon seeing the Byzantine troops advance upon him, he fled in such haste that he left his crown and sceptre behind him.