It was located in what is now part of the city of Sano, Tochigi Prefecture, in the northern Kantō region of Japan.

Fujiwara no Hidesato was assigned rule over Shimotsuke Province following the suppression of the revolt of Taira no Masakado in the Tengyō no Ran, and his descendants controlled the castle for five generations until the clan relocated its seat to Ashikaga.

During the Sengoku period, the northern Kantō region was continually contested between the powerful Uesugi and Hōjō clan.

Following the Siege of Kawagoe Castle in 1546, Uesugi Kenshin made strong inroads into the region, forcing many of the smaller warlords to submit to him.

However, Kenshin's lines were overextended from his base in Echigo province and he faced continual threat of attack along his flank from his arch-nemesis, the Takeda clan.

The geography makes this tale impossible; however, it is likely that the Shogunate was uncomfortable with having the stronghold of Karasawayama remain in the hands of a clan who had historically showed a tendency to switch sides at the least opportunity.