The lower portion of Shiroyama contains the Hida Sites, the ruins of a Yayoi period fortified settlement, which indicates that the location was seen as a strategically important place for over 1800 years.

Shingen's arch-rival, Uesugi Kenshin had his capital at Kasugayama Castle in what is now the city of Jōetsu, only 50 kilometers from the border with Shinano.

The inner bailey consisted of several terraces approximately 50 meters long, protected on its north edge by steep dry moats.

The entrance was to the east, which was a kuruwa with clay and earthen ramparts, dry moats and terraces intermittently extending along a long ridge.

South of the inner bailey was the third area, with smaller terraces built up the steep hill in steps, and formed the rear of the castle.