This is in the Hitoyoshi Basin, a 30 by 15 kilometer valley surrounded by tall mountains at the southern border of Higo Province.

A Shinto shrine, the Sagara Gokoku Jinja is now located on the west side of the castle ruins.

The Hitoyoshi City Hall was formerly located within the castle, but was relocated after the damage caused by the 2016 Kumamoto earthquakes.

However, after an internal dispute over the succession issue, on July 14, 1526, Hitoyoshi Castle was surrounded by a large army led by the Kitahara clan (who were backed by the Ikkō-ikki movement).

[3] After that, the 19th head of the family, Sagara Yoshiharu, began a major renovation of the castle during the Tenshō era (1573-1593), which was only completed in 1639.

In 1587, Sagara Yorifusa fought bravely during Toyotomi Hideyoshi's conquest of Kyushu, but surrendered, and was granted Hitoyoshi Castle and territory as an independent lord again through negotiations with his vassal, Fukamizu Nagatomo.

In the Battle of Sekigahara in 1600, he initially sided with Ishida Mitsunari's (Western Army) and attacked Fushimi Castle and other castles, but after Ishida's defeat, he defected to the Tokugawa (Eastern Army) and was granted confirmation in his own holdings with an assessed kokudaka of 22,000 koku.