Middelburg was founded by Pieter Bladelin, a high-ranking official responsible for financial matters in the court of the Duke of Burgundy, Philip the Good.

[1][2] Bladelin had begun purchasing parcels of land in the area already in 1433, and in 1444 his holdings were united into a single fief and manor.

However, the castle was again captured and partially destroyed by troops from Bruges in their struggle with the future Emperor Maximilian in 1488.

During the Eighty Years' War it again changed hands several times and reduced to a state of ruin, after which it began to be dismantled by locals using it as a quarry.

[1] It contains the elaborately carved tomb of Pieter Bladelin and his wife, Margaret van de Vageviere, in addition to a polychrome altarpiece, a Rococo pulpit and a Baroque confessional.