The first recorded lord of Pampigny is mentioned in 1228 as a vassal of Jean de Cossonay.

Around 1400 Pampigny and the castle were acquired through marriage by the Moudon family and sold in 1439 to Jean de Menthon, the lord of Aubonne.

While the date of construction of the second castle is unknown, the Late-Gothic style church indicates that it happened before the Renaissance reached the area.

In the late 17th century the Mestral family built the third castle[2] which incorporated the wall with windows from the second.

After the 1798 French invasion of Switzerland, the nobility lost much of their land and power though the family retained the castle.