This cadet branch was named the Tō clan (東氏) and due to their skill at waka poetry, became close retainers of the Ashikaga shogunate in the Muromachi period.

However, conditions became increasing difficult in the Sengoku period, and the Tō were barely able to repulse an invasion by the Asakura clan from the north.

However, after the Toyotomi forces defeated Shibata Katsuie and Oda Nobutaka, Endo Yoshitaka surrendered, and Hideyoshi awarded the castle to his retainer, Inaba Sadamichi.

Both rivals fought on the same side during the Battle of Sekigahara in 1600 and Tokugawa Ieyasu resolved the dispute by awarding Inaba Sadamichi with Usuki Domain in Kyushu and returning Gujō Hachiman Castle to Endō Yoshitaka.

The Endō clan ruled for a century, but declined due to internal conflicts and was transferred to a smaller territory in the Kantō region in 1693.

In accordance with directives issued by the Meiji government, all remaining structures of the castle were demolished in 1870, leaving only the inner moats and some stone walls.