Although most of the building was built during the Renaissance period, on the site of a far older fortress, it gained some slightly Neoclassical features after its renovation during the 18th century.
According to legend, Baldwin Iron Arm, Count of Flanders, kidnapped Judith, the daughter of Charles the Bald in 862 in Senlis and brought her to the fortress that used to be at the very location where the present castle now stands.
With the intervention of the Pope, the two lovers married in Auxerre, which then resulted in the creation of the County of Flanders.
In fact, during World War I, it was taken over by the Germans and became for a short period of time, the home of Baron Manfred von Richthofen, better known as the Red Baron, who was credited with shooting down 80 enemy aircraft in his brightly painted red airplane.
The castle can be seen in the movie That Lucky Touch, a comedy which was partially shot in Rumbeke in 1975, starring Roger Moore and Shelley Winters.