The area develop from the Heian period as one of the main ports connecting Shikoku with the Kinai region of Japan.

However, by the 1570s, the Miyoshi were defeated by Chōsokabe Motochika, a minor warlord from Tosa Province who succeeded in uniting most of Shikoku under his rule.

Sanuki Province was then assigned to Hideyoshi's general Ikoma Chikamasa, who initially made Hiketa Castle his stronghold.

Following the establishment of the Tokugawa shogunate, the government issued a decree in 1615 limiting the number of castles permitted by each domain to only one.

The castle once consisted of several enclosures spread across the ridge of the Shiroyama hill overlooking Hiketa port for a distance of 400 meters.