The site was the location of a burial mound which was believed to have been the grave of the Nara period poet Kakinomoto no Hitomaro.

The location dominates the San'yōdō highway connecting the Kinai region with western Japan and also the main route north to Tanba and Tango Provinces.

Considered by the Tokugawa shogunate to be a backup to Himeji Castle, it was the final line of defences for the Kansai region against any attack from the west.

[2] The construction of the castle was by order of Shogun Tokugawa Hidetada from 1617 to 1619, who had assigned Ogasawara Tadazane to the area as daimyō of the newly created 100,000 koku Akashi Domain.

It is claimed without clear documentary evidence that Miyamoto Musashi assisted in building the castle as a "Construction Supervisor", as it is recorded that he was in the service of Ogasawara Tadazane at the time.