The original château was constructed in the 17th century by Marie de Bourbon, Countess of Soissons and Princess of Carignano after her marriage to Prince Thomas Francis of Savoy.

François Le Juge owned the property until 12 March 1719 when Philippe d'Orléans, (Regent of France during the minority of Louis XV), acquired it.

She had the architect Claude Desgots (nephew of André Le Nôtre) add two large wings to the building; she also had the park redesigned.

In 1734, the Dowager Duchess of Orléans[5] let her daughter Philippine Élisabeth d'Orléans stay at the château; the young woman had once been engaged to the future Charles III of Spain; the couple was much in love, but the marriage never took place.

At the death of the Duchess on 1 February 1749,[6] her son, Louis d'Orléans, Duke of Orléans, inherited the Bagnolet estate.

The Hermitage is the only structure from the old estate to exist today; it can be found at 148 rue de Bagnolet, in the 20th arrondissement of Paris.