His great-niece Denise de Fougère, having married Sulpice d'Amboise, passed the château into the Amboise family for five centuries.
Pierre d'Amboise unsuccessfully rebelled against King Louis XI and his property was confiscated, and the castle was dismantled on royal order in 1465.
In 1594, at the death of Diane's granddaughter Charlotte de la Marck, the château passed to her husband, Henri de La Tour d'Auvergne, Duke of Bouillon, who sold it to a tax farmer Largentier, who had grown rich on gathering in the salt tax called the gabelle.
His eventual heir was forced to sell Chaumont to pay his debts to a maître des requêtes ordinaire to Louis XV, Monsieur Bertin, who demolished the north wing built by Charles II d'Amboise and the Cardinal d'Amboise, to open the house towards the river view in the modern fashion.
[citation needed] However, in 1789, the new French Revolutionary government seized Le Ray's assets, including his beloved Château de Chaumont.
By 1851 the "Chaumont suite"[7] of the early-16th century Late Gothic tapestries with subjects of country life emblematic of the triumph of Eternity, closely associated with Chaumont and now at the Cleveland Museum of Art, was still hanging in the "Chambre de Catherine de Médicis"; the tapestries had been cut and pieced to fit the room.