The Fortezza is built on a hill called Paleokastro (meaning "Old Castle"), which was the site of ancient Rhithymna's acropolis.

After Crete fell to the Republic of Venice, the settlement became known as the Castel Vecchio or Antico Castello, which both mean "old castle.

"[2] Under Venetian rule, a small harbour was built in Rethymno, which became the third most important city on Crete after Heraklion and Chania.

These fortifications were not strong enough to withstand a large assault, and when Uluç Ali Reis attacked in 1571, the Ottomans captured and sacked the city.

The new fortress, which was built on the Paleokastro hill, was designed by the military engineer Sforza Pallavicini according to the Italian bastioned system.

The fortress was built under the master builder Giannis Skordilis, and a total of 107,142 Cretans and 40,205 animals took part in its construction.

[2] Although the original plan had been to demolish the old fortifications of Rethymno and move the inhabitants into the Fortezza, it was too small to house the entire city.

A number of buildings are located within the Fortezza, including: The fortress also contains an armoury,[19] two gunpowder magazines,[20] storage rooms[21] and several cisterns.