Taking advantage of his absence, Mōri Motonari, a small lord from Aki Province attacked and defeated the Sue at the Battle of Itsukushima in 1552.

However, after the 1600 Battle of Sekigahara, the Mori clan were deprived of two-thirds of their territory by the Tokugawa shogunate, including Iwami Province.

As retainers of the Mōri, this meant that the Yoshimi clan also lost their estates in Tsuwano and was forced to relocate to Hagi.

Sakazaki laid the foundations of Tsuwano Domain by building a castle town, conducting land surveys, and making major renovations to the castle, including building 15-meter high stone walls to enhance its defenses.

The original Tsuwano Castle was a typical Muromachi-period fortified mountain, with enclosures spread out across a two-kilometer ridge extending north and south.

In the central enclosure, he build a tenshu which was probably three-stories on a 20-meter square foundation and added numerous yagura turrets and compound gates.