The first traces of the castle appeared in 830 with the construction of a castrum by the abbot Hilbold, from the monastery of Saint-Philbert.
The castle was rebuilt in stone in the 12th century by the feudal power who was trying to stabilise the region, notably by preventing Norman pillaging.
The castle resisted numerous attacks: but in 1674, it was taken by the Dutch troops of Admiral Cornelis Tromp.
The rectangular fortification consists of two towers, a single gate and two watch turrets in the four corners.
Property of the commune, the Château de Noirmoutier is listed as a monument historique by the French Ministry of Culture.