From around the 1460s until their conquest by Oda Nobunaga in 1581, these warrior families, later known as ninja, self-governed the province as the Iga ikki, with a headquarters at the site of where the castle now stands.

Under Toyotomi Hideyoshi, Tsutsui Sadatsugu (1562-1615), was transferred from Kōriyama Castle in Yamato Province to rule over Iga.

Following the Battle of Sekigahara, Tokugawa Ieyasu expelled Tsutsui Sadatsugu from Iga Province for his pro-Toyotomi stance and installed his trusted general Tōdō Takatora to rule over Iga and the northern half of Ise Province.

He also significantly expanded the castle area overall, with large kuruwa terraces of over 100 meters in length extending to the east and west.

However, after the Toyotomi clan was extinguished at the Siege of Osaka in 1615, the castle lost its importance and the tenshu (along with a planned series of water moats and stone walls) was never completed.

[5] Parts of the Akira Kurosawa movie Kagemusha were filmed at Iga Ueno Castle.