At the instigation of Louis XV, Charles François Paul Le Normant de Tournehem, directeur général of the Bâtiments du Roi, negotiated the purchase of the land from its various owners, located between the royal châteaux at Meudon and Saint-Cloud.
In 1749, Louis gave the land to Madame de Pompadour, who delegated the building to her architect, Jean Cailleteau (called "Lassurance the younger"), assisted in the gardens by Jean-Charles Garnier d'Isle.
It was a relatively modest building, more like a house of a rich nobleman than a royal château, reflecting its role as an intimate meeting place for King Louis and Madame de Pompadour.
In 1750, Madame de Pompadour acquired a small building at the bottom of the slope, on the banks of the Seine, called Brimborion (French: trinket), which was linked to the new residence through its gardens.
After the King and Madame de Pompadour grew apart, Louis repurchased it in 1757 and had it redesigned by Ange-Jacques Gabriel, adding two wings linked to the main building.