In 1024, he submitted to Pilgrim, Archbishop of Cologne, when the latter was besieging Capua on behalf of Emperor Henry II, though his own duchy was not threatened.
Basil Boioannes, the Greek catapan of Italy, negotiated a surrender and gave Pandulf V safe conduct to Naples, where Sergius offered him asylum.
In the next year (1027), after Sergius' ally Boioannes was recalled, Pandulf attacked Naples and quickly captured it, some say by treachery.
Sergius and the Duke of Gaeta, his nephew, John V, sent an embassy to the Norman to ask his assistance in regaining the Neapolitan duchy.
Broken in spirit, Sergius retired to the monastery of the Holy Saviour in insula maris, where the Castello del'Ovo now stands.