It was the primary fortress of the warlord Uesugi Kenshin, and was originally built and ruled by the Nagao clan.

The castle had no stone walls but consisted of many enclosures (kuruwa) with earthen or clay ramparts and dry moats built on terraces at various levels on the mountain.

The core of the castle (or inner bailey) was just below the peak, and contained a watch tower, Buddhist temple (and Uesugi Kenshin's residence) and garden.

Hideharu built a large water moat to surround the jōkamachi of Kasugayama as the local populace was highly upset over the departure of the Uesugi, and in 1600 he relocated his seat to Fukushima Castle near the coast and port of Naoetsu.

The ruins of the inner bailey are located 180 m.a.s.l., with view at the city of Jōetsu, Kubiki Plain, and the Sea of Japan, approximately 20 minutes on foot from the shrine.