In 1584, during the early part of the Eighty Years' War, the castle was burned down by bandits.

At that time the castle was owned by Jacob de Bryarde, and it remained in the Bryarde family till 1828, by which time the family fortunes were in decline and the castle was falling into a state of increasing disrepair.

The building was expanded under plans drawn up by an architect from Veurne called Jozef Vinck.

Merghelynck died childless in 1905 and the castle was bequeathed to the Belgian state on condition they made it available to the (subsequently renamed) Royal Academy of Dutch language and literature (KANTL).

[1] In 1998 responsibility for the castle was transferred to a regional level, coming under the remit of the Flemish Heritage foundation.