Although there is little documentary evidence, according to local folklore, following the Former Nine Years' War, Minamoto no Yoshiie was awarded Tajima Province, and he ordered the construction of Yagi Castle in 1063.

Initially, the Yagi cooperated with the Mōri, but in 1579 surrendered to Nobunaga's general, Toyotomi Hidenaga in exchange for being allowed to keep their castle and territory.

Consequently, Toyotomi Hideyoshi seized Yagi Castle and awarded it to Bessho Shigemune as part of a 15,000 koku fief.

[2] Bessho Shigemune rebuilt the castle into its current configuration, with a new inner bailey with connecting kuruwa enclosures on all sides, yagura watchtowers, and stone walls lining the fortifications facing the jōkamachi, which had grown up along the route of the San'in highway.

In 1628, he was dispossessed of his domain for negligence in attendance to his duties at Edo Castle, which may have been a pretext for the shogunate to seize control of a gold mine located on his territories.