Dating from the end of the 17th and beginning of the 18th century, it was profoundly transformed by the Duke of Castries between 1843 and 1863, under the architects Joseph-Antoine Froelicher and François Clément Joseph Parent.

The hôtel was originally built around the end of the 17th century by Jean Dufour, seigneur de Nogent.

A 1761 inheritance from his uncle the Marshal of Belle-Isle left Charles to with a huge fortune, and allowed him to initiate grand works on the hôtel's interior decoration, led by the sculptor Jacques Verberckt.

A new gate onto the street was built by Jacques-Antoine Payen in 1762, which may still be seen, and the two wings were linked by a high wall surmounted by a balustrade.

On his death in 1886, his widow remarried, to vicomte Emmanuel d’Harcourt, who sold the hôtel de Castries to the Montgermonts.