The oldest still existing parts date from the 12th century and were built by Manassès de Seignelay (bishop from 1207 to 1221).

Still standing is the main rectangular plan building, flanked by three towers, a fourth having been destroyed.

The guard room, the lower hall with ogive vaults and the cellars are the last elements which constituted the 13th-century castle, the episcopal palace at that time.

During the Hundred Years' War, the building was transformed into a fortress; it was taken from the English by Joan of Arc on 14 June 1429.

The castle was abandoned from the Wars of Religion until the start of the 18th century when Bishop Fleuriau d'Armenonville undertook the transformation of the structure into a comfortable residence.