The exact year of construction is not known, but the village of Padern is recorded in 899, when Charles III, known as Charles the Simple, ceded the territory to Lagrasse Abbey, while the latter still belonged to the Counts of Toulouse, and not to the crown of France.
After the capture of Quéribus, he negotiated his freedom for the abandonment of the citadels to King Louis IX.
The Abbey of Lagrasse thus recovered the castle, and in 1283, became the official proprietor through a transaction with Philip III of France (Philippe the Bold).
At the end of the 16th century, after the Wars of Religion, Pierre de Vic, originally from Girona (Catalonia), acquired the castle and made some additions, though conserving its feudal aspect.
In 1706, his descendants resold the whole property to the Abbey of Lagrasse, which abandoned it at the end of the 18th century.