According to later tradition, Emperor Alexios II Komnenos sent twelve noble families to Crete, in order to strengthen the ties between the island and Constantinople.

As Richard von Coudenhove-Kalergi describes in his book An idea conquers the world the Kallergis name is composed of the Greek word kallos (=beauty) and ergon[5] (from ergō="work, task, deed, accomplishment, or purpose") [Greek: Καλλ(ι)έργης > Καλλέργης, known in many versions as Kalergis, Calergis, Kallergi, Callergi, Calergi].

Heraldic ensembles containing the Kallergis family coat of arms (bendy argent and azure) can be found all over the island of Crete, in churches and other monuments.

[7] Their prominent position and privileges survived during the Venetian dominion of Crete as they were part of the "privilegiati" (Greek: Αρχοντορωμαίοι) and sometimes of "nobili Veneti".

Victor Callergi bought the famous building on the year 1589 for 36,000 ducats, and now belongs to a Venetian branch of the family which gave its name to it.