The towers were named by Herod after his brother Phasael, his friend, general Hippicus, who had fallen in battle, and his favourite wife, Mariamne.

When the city was razed in AD 70, all three towers were left standing, in order to show off the strength of the fortifications the Roman army had to overcome.

In medieval times, Muslim Arabs called the gate next to the tower Bab Mihrab Dawud, or "Gate of David's Chamber or Sanctuary", on the belief that the room atop the Herodian tower stump represented the "private chamber" or "prayer room" of Prophet Dawud (King David), which is specifically mentioned in the Quran (Surah 38.

[7] The Citadel was gradually built up under Muslim and Crusader rule and acquired the basis of its present shape in 1310, under the Mamluk sultan Malik al-Nasir.

These include part of the Hasmonean city wall, a Roman cistern, and the ramparts of the Umayyad citadel, which held out for five weeks before falling to the Crusaders in 1099.