In the 12th century the lords of Heusden had a territory which stretched on both sides of the Oude Maasje.

In the part south of the Oude Maasje was the main village Oudheusden with a church.

In 1130 the main castle of the Lord of Heusden was near this church, on the left bank of the Oude Maasje.

The new settlement was also east of a north–south water which connected the Oude Maasje to the Meuse.

Jan van den Elshout then received the castle back as a personal loan.

[3] After the death of Jan V in 1330, the claims of the Brabant dukes as Lords of Heusden became indisputable.

John III, Duke of Brabant therefore started to invest in its expansion as a border fortress against Holland.

[6] In the first half of the 16th century Heusden was in the front line of the Guelders Wars (1502-1543).

In the beginning of the Eighty Years' War Kapitein Waardenburg took Heusden for the rebel side in 1572.

The loyalist commander Spiering van Wel retreated to the castle.

A gunpowder accident then caused a fire that would later become known as the Grote stadsbrand van Heusden, leaving most of the city in ashes.

It made such an impression that Spiering van Wel left the town to the rebels.

The castle was almost completely destroyed, and of the donjon only a crater filled with black water was left.

Only the high square tower which survived the blast more or less intact was left (see 1691 drawing).

In its center, the circular main castle had an octagonal tower with a diameter of 8 m. On the northeast side was a square building of 10 by 10 meters.

From the outside of the circle, ground had been added till a level of at least more than 2 m high around the octagonal tower.

[10] The octagonal tower had walls which were 2 m thick at ground level, and were made of Meuse limestone.

[12][13] For Jaap Renaud, the leader of the excavations, the use of stone instead of brick was reason to date it to before 1200.

It revealed a plan to let a consortium of real estate developer Amvest, Schakelring Heusden and Service Residentie Molenwijck redevelop the ruins.

It threatened the monument, and could hardly be reconciled with the legal status of Heusden as a Beschermd Stadsgezicht (protected cityscape).