Shimo-Akasaka Castle is located on Mount Kabutori, a ridge extending from Mount Kongō on the border of Kawachi Province with the Yoshino region of Yamato Province at an elevation of 185.7 meters above sea level and 61.4 meters above its surroundings.

It was part of a defensive position which included many smaller fortifications on surrounding hills, and later, Kami-Akasaka Castle on the hillside opposite a small river.

In 1331, when Emperor Go-Daigo attempted to overthrow the Kamakura shogunate in the Genkō War, he relied on Prince Moriyoshi and a local lord from the Chihaya-Akasaka area, Kusunoki Masashige, to raise an army.

However, this first attempt failed, and Emperor Go-Daigo was captured and exiled to the Oki Islands.

In early 1333, Kusunoki Masashige joined forces with Prince Moriyoshi and recovered Shimo-Akasaka Castle and resumed his offense against the shogunate.