The first castle of Gaillon belonged to a whole system of defence along the Norman border such as Evreux, Pacy-sur-Eure, Vernon, Malassis, Gasny, Baudemont, etc.

In 1192 King Philip II Augustus of France seized the castle in his battle with Richard the Lion Heart to conquer Normandy.

Richard decided to build a new one a few kilometers away in Les Andelys on the other bank of the Seine River : Château Gaillard.

Five years later John Plantagenet, Duke of Bedford, ordered the demolition of all fortifications, sparing only the home of the archbishop.

A new castle was built around 1500 - 1509 by Georges d'Amboise, Archbishop of Rouen and Minister of Louis XII in the modern Renaissance style recently imported from Italy.