Facing the threat of the Hundred Years' War, the lords of Châteauneuf then built the powerful fortifications around the 12th century keep.
He then modified the castle to make it more comfortable as a residence, in keeping with the style of the court of the Duchy of Burgundy.
A chapel and a residence dated to this time were built out in the court of the castle, in the flamboyant Gothic architectural style.
Philippe died in 1493 without leaving an heir, so the castle went to his brother Guy Pot, then via various alliances to Marie Liesse de Luxembourg.
With their entrance into the convent, the castle then passed to Charles de Vienne, count of Comarrin, and remained in the hands of this family for the next 150 years.