The Kasbah was built on top of a hill 305 meters tall in northern Béja, This site was chosen so the castle can overlook the city and its surrounding plains and countryside.
Because of that, Carthage needed to protect Vaga (especially from barbarian raids), but after the decline of the Carthaginian power, especially after the third Punic war, the city fell under the influence of the Numidians and their king Jugurtha made the fortress his center of reign.
In 943 the Kasbah suffered major damage when Béja was sacked by the Kharijite rebels, led by Abu Yazid, and was restored by the Fatimid Caliph al-Mansur bi-Nasr Allah in 946.
After the independence of Tunisia from France, the fortress was turned into a cultural center, but today the Kasbah is in bad conditions.
Restorations started in 2005 and directed by the Tunisian National Heritage Institute, but it has stopped for lack of funding.