The Historical Site designation also includes the Taikōganaru (太閤ヶ平) fortification erected by Toyotomi Hideyoshi during the 1581 Siege of Tottori.

[2] It is claimed that in the late 12th century, following the Genpei War, the new Shōgun Minamoto no Yoritomo granted the position of shugo of Inaba Province to Nasu no Yoichi, the hero of the Battle of Yashima.

The Amago were defeated by the Mōri clan of Aki Province in 1566 and Inaba Province became a battleground between the Mōri, the remnants of the Amago, and the waning power of the Yamana, who under , Yamana Toyokuni moved clan's main stronghold to Tottori castle from Tenjinyama castle in 1573.

He also built a semi-circular 12-kilometer long encampment called the Taikoganaru (太閤ヶ平) on three sides of the castle to isolate it from the town and any supplies or reinforcement.

The mountain areas of the castle, including its original inner bailey were abandoned, but at the lower levels a three-story tenshu was constructed.

Little remains of the castle aside from parts of the stone wall and one gate, reinforced with iron and featuring spikes on the outside of the doors to help protect against attackers.The site became a park and grounds for a high school.