[2] A fortification was originally built on a nearby mountain (Mount Omatsu) in 1240 AD by Akiba Shigenobu, but its details are not clear.

The location is of strategic importance in controlling the north-south transportation route between the Seto Inland Sea and the San'in region on the Sea of Japan, and the east-west transportation route between Tsuyama in Mimasaka Province and Miyoshi in the center of the Chūgoku region.

[5] His son, Mimura Motochika recaptured the castle in 1571[5] and greatly enlarged it, extending the site to cover the entire mountain.

[5][6] In 1600, following the Battle of Sekigahara, the Mōri clan were deprived of Bitchū Province and were forced to cede the castle to Tokugawa Ieyasu.

After his son died without heir in 1641, the domain went to Mizunoya Katsutaka, who rebuilt the tenshu, yagura turrets and gates in addition to rebuilding the castle town.

Three structures were saved and still stand today: a short section of wall, the Nijū yagura, and the tenshu.

The presence of Sanjūrō, who was first made a provisional mascot before being named as "lord", has helped increase the number of visitors to the castle.