In the Roman era, the main road that linked up Fréjus to Riez passed through Vérignon, and four milestones have been discovered.

According to Raoul Bérenguier, the Blacas seigneurs of Aups installed themselves in Vérignon around the year 1000 and constructed their castle on a rocky outcrop in the town.

It had an encircling wall, rectangular living quarters in the north corner, a private chapel of the Assumption for the Blacas (with their family vault under its altar), and a barbican round its south gate.

However, this castle proved too constricted, uncomfortable and small, and was abandoned at the start of the 18th century, with the Blacas building a new, unfortified château, which remained in their hands until 1947.

A lady from the Blacas family came here to await the arrival of her son back from the Crusades and vowed to construct a chapel as soon as he should return safely.