After the Battle of Uji (1180), the head of Minamoto Yorimasa was brought to Koga by the Shomokobe clan.

A Shinto shrine to Minamoto Yorimasa still exists at the present Koga Castle site.

His descendants ruled to 1583, when the castle was taken by the Later Hōjō clan, in the same year Ashikaga Ujinohime took the title Koga kubō.

Due to changes in the flow of the Watarase River and flood control work from 1910, almost nothing remains of the castle today.

An additional forecourt, the Suwa Kuruwa (諏訪輪 ) occupied a small area on the opposite bank of the river.