After his release, he took refuge with his uncle the Cardinal de Périgord in Avignon, which had just been sold to the Papacy by Joan I of Naples.
(Robert had married, by force, Maria of Calabria; but falling into the hands of Louis of Taranto, was murdered by his wife's orders in the Castel dell'Ovo.)
Robert was not released until 18 March 1355, through the efforts of his uncle and Pope Innocent VI, and was made to swear to take no revenge on his captors.
After an unsuccessful attempt at mediation by Walter VI of Brienne and others, local levies besieged the castle, and by July, Robert was compelled to surrender it to its rightful owner.
This violation of the neutrality which ought to adhere to a churchman and mediator so provoked Edward the Black Prince that he had Robert's body borne on a shield to the Cardinal as a mocking salute.