The castle was destroyed, however, by Brussels city troops in revenge for the assassination of Everard t'Serclaes, which was allegedly commanded by the Lord of Gaasbeek at the time, Sweder of Abcoude.

At the beginning of the 16th century, the Dominium of Gaesbeeck was inherited by the House of Hornes; they constructed a brick castle on the ruins of the medieval fortress.

In 1565, Lamoral, Count of Egmont, acquired the castle and its domain, including feudal rights in 17 surrounding villages.

Accused of high treason by Philip II of Spain, the Count of Egmont was beheaded three years later on the Grand-Place/Grote Markt (main square) of Brussels.

The works were executed by the architect Charles Albert and ordered by the Marquise Arconati-Visconti (née Marie Peyrat) who owned the castle at that time, having inherited it from her husband.