Famous as the birthplace of Lady Kasuga, the wet nurse of Tokugawa Iemitsu,[2] it ruins have been protected as a National Historic Site since 1989.

A fortification was originally built in this location in the early Muromachi period by the Akamatsu clan, who ruled the area for around 120 years.

Known as the "Red Demon" of Tanba, Akai Naomasa seized Kuroi Castle in 1554 by killing his grand-uncle, and over the next ten years expanded his holdings through marriage alliances, open warfare and stratagem to control the western half of the province.

[2][4][5] Mitsuhide placed his general Saitō Toshimitsu in charge of Kuroi Castle, and as the area as still threatened by Tajima Province, he had the defenses modernized with stone walls and extended.

[1] In 1580, Hashiba Hidenaga, the younger brother of Toyotomi Hideyoshi seized Tajima Province, expelling the Yamana clan and eliminating the threat to Kuroi Castle.

Toshimitsu's daughter, Fuku (1579-1643), was born at the Saitō residence at Kuroi Castle, survived and married with Inaba Masashige.