After Philip's death, Charles V, Holy Roman Emperor confiscated all territorial possessions of the bishopric of Utrecht, including Duurstede Castle.

The states, however, invested their money into building modern fortification around Wijk bij Duurstede, and as a result the castle fell into neglect.

Further damage was done when French troops devastated Wijk by Duurstede in 1672, after which the townspeople used stone from the castle to rebuild their homes.

The old donjon built by Zweder van Abcoude in the 13th century has withstood the passage of time reasonably well, and is an excellent example of medieval towers.

The walls are 2.5 metres thick; the original entrance was at the second level and could only be reached with a wooden ladder that could be removed or destroyed in times of need.