The fort is located on the top of a mountain rising 236 meters above sea level in the north of the town of Agadir near the current port.

[6] Prior to the fortification of the site by the Sa'dis, the Portuguese nobleman João Lopes de Sequeira occupied the area in 1505.

[7][8][9][10] He built a wooden castle at the foot of a hill[8] and a Portuguese colony named Santa Cruz do Cabo do Gué was created.

[8] Their presence elicited growing hostility from the local population of the Sous, until in 1540 the Sa'di sultan Muhammad al-Shaykh occupied the main hill above the city and installed artillery to prepare an attack on the Portuguese fortress below.

[8] The site was then left unoccupied for years until Muhammad's successor, Abdallah al-Ghalib (r. 1557–1574), built a new fortress on the hilltop.