In 1336, Kitabatake Chikafusa built a castle at this location initially to protect the eastern approaches to Yoshino, where the emperor had found refuge, but also access to the important shrines at Ise and Kumano.

Nobukatsu commissioned the addition of stone walls and a tenshu, which were completed just before Nobunaga's construction of Azuchi Castle, which is also known to have featured a tower.

After Nobunaga's assassination in 1582, he was briefly allied with Toyotomi Hideyoshi, but was deprived of his territories and in the end survived only a minor daimyō within the structure of the Tokugawa shogunate.

After the start of the Tokugawa shogunate, the castle was awarded to Inaba Michito as the center of a 45,700 koku Tamaru Domain.

After being transferred to Fukuchiyama Domain in 1624, he went insane, and after murdering 60 townsmen, was ordered to commit seppuku by the shogunate.

The tenshu was located at the northern edge, and a 25-meter square foundation base remains.

To the south was the Ni-no-maru secondary enclosure, which was separated by a dry moat and bridge, with Masugata-style gates.