The first castle was built by Robert I, Count of Flanders, at the end of the 11th century and used as a base for military operations.

Adolph of Cleves, Lord of Ravenstein, and his son Philip of Cleves-Ravenstein transformed the castle into a beautiful mansion.

In 1482 Mary died of a fall from her horse here, an accident which changed the history of the Low Countries: under her authoritarian husband Maximilian I, Holy Roman Emperor, a period of more than 300 years of Habsburg rule began.

In the second half of the 16th century, the Dukes lost interest in their Flemish possessions and in 1578 part of the castle was burned down by Protestants.

In 1609 Duke John William of Cleves died without children and after the War of the Jülich succession in the Treaty of Xanten, the Fiefdom of Wijnendale passed to one of the victors, Wolfgang Wilhelm, Count Palatine of Neuburg, as confirmed in 1634 by the Privy Council of the Habsburg Netherlands and in 1666 by the Treaty of Cleves.

In the middle of the 18th century, Duke Charles Theodore constructed roads in West Flanders to improve trade, with Wijnendale in the center.

His son Joseph Louis Matthieu modified it in 1877 and gave it its present romanticized, mediaeval form.