Even though it was originally built on a river, the Van Herlaers believed that all four sides of the castle needed to be fortified.
To add to the security, the original design included a moat that ensured any attacker could only invade through the front of the fortress.
Given that the village was built on a river with very arable lands, it was most likely just a small farming community with little importance until the creation of the castle in the 14th century.
Despite his close relationship with Duke William, Arent Hoeman sided with the Duchess of Brabant.
William, unsurprisingly, saw this as an act of betrayal and abruptly ended his friendship with Arent Hoeman.
Just days later, Duke William invaded Ammerzoden, capturing both Arent Hoeman and the castle.
His son did not hold the castle for long, as he sold it to Johan Van Broeckhuysen, lord of the Waardenburg, in 1424.
Burgundian troops, led by Count Henrick Van Nassau, occupied the castle for a brief time in the mid-16th century.
During the siege and subsequent occupation, the castle was moderately damaged and a large amount of personal property needed to be replaced.
The lord of the castle at the time, Joris Van Arkel, died from injuries that he sustained during the fire.
This fire took place during the 80 years war and caused the family to have very uncertain feelings about its economic future.
When the Van Arkels repaired the castle they got rid of the grand hall, elected to have numerous, smaller rooms instead.
Because the original Castle was built primarily for defense, there were not many windows as they made the fortress more vulnerable during attacks.
Not long after the rebuilding was completed in mid-17th century, King Louis XIV of France invaded the Netherlands.
By the time the French soldiers arrived at the castle in 1672, the Van Arkels were able to regain some of their prior wealth.
In 1693, following the death of the last Van Arkels, Thomas, the castle came into the hands of a number of other Dutch families over the next century.
The start of World War II led to the end to the convent as the area was bombed heavily.
Among the pieces found were fragments of pottery, stoneware, glass, silver, bronze, and tin.
From 1976 until 1999 the castle was used not only as a tourist attraction, but also as an official town hall and meeting place for the local municipality.