[1] The oldest part of the castle, the southern residential tower, may date to the late 12th century.

It may have been built for a local noble, Albertus de sancto Victore, who is mentioned in 1168,[2] or for the Counts of Sax.

[4] The two towers were separated by a three meters (ten foot) wide gap and were linked by a bridge.

Around 1400 the northern tower was raised to its present height of six stories and topped with a gable roof.

[5] The two towers that make up the castle were built on a small terrace on a mountain north of the village of San Vittore.