However, during the Battle of Sekigahara in 1600, the Tomita clan sided with the eastern forces under Tokugawa Ieyasu and Tsu Castle was attacked by the western armies under Mōri Terumoto and Chōsokabe Morichika.

The Tomita were rewarded for their loyalty by the Tokugawa shogunate and were given an increase in kokudaka and rebuilt parts of the castle by the time they were transferred to Uwajima Domain in Iyo Province in 1608.

The Tomita were replaced by the Tōdō clan, who ruled as daimyō over the 220,000 koku Tsu Domain until the Meiji Restoration.

After the tenshu was destroyed in a fire in 1662, the Tokugawa shogunate did not grant permission for it to be rebuilt, and it was replaced by a two-story yagura.

However, it is not a historically accurate reconstruction, and contains various features copied from other structures to make it more visually appealing.

The gate to the Japanese garden now located within the Inner Bailey is a survival from the Edo period han school of the domain.