The location was formerly surrounded by inlets, which enhanced its defensive position, and its ability to extend control over shipping in the area.

In 1480, Toda Munemitsu (1439–1508), virtually independent warlord of the Atsumi Peninsula during the Sengoku Period, erected the predecessor of Tahara Castle.

When Matsudaira Hirotada was forced to send his son, the future Tokugawa Ieyasu to Sunpu as a hostage to the Imagawa, he turned to the Toda clan for assistance, but the Toda sent Ieyasu as a hostage to Oda Nobunaga instead, in exchange for a monetary payment.

Following the establishment of the Tokugawa shogunate, Toda Katatsugu was raised back from hatamoto status to a 10,000 koku daimyō, and allowed to return to Tahara Castle, which was now the administrative center of the newly created Tahara feudal domain in 1601.

In 1664, his son Toda Tadamasa was transferred to Amakusa Domain in Bungo Province with an increase in revenues to 21,000 koku and Tahara Domain was reassigned to the Miyake clan, who remained in residence until the Meiji Restoration.