It was originally built on the seashore, with the ocean forming a natural moat on three sides; however, due to land reclamation it is now in the center of the city.

During the Muromachi period, a branch of the Saionji family was appointed as governor of the area by the Ashikaga shogunate, but was constantly being invaded his more powerful and aggressive neighbors.

The Saionji survived by the fluid loyalties and fierce resistance, but were eventually overcome by Chōsokabe Motochika, who was in turn overthrown by the forces of Toyotomi Hideyoshi.

Takakage was later transferred to Kyushu and was replaced by Hideyoshi's general Toda Katsutaka, who expanded on the minor fortification, transforming it into a Sengoku period castle.

Tōdō Takatora retained the original layout of the castle, which had an oblong inner bailey measuring 30 by 100 meters, with the tenshu in the center of the southern portion.

During the Bakumatsu period, Date Munenari introduced military reforms and built up a western-style army; however, the domain did to participate in the Boshin War and the castle was undamaged.

The castle initially escaped the effects of government decrees for the destruction of all former feudal fortifications, but from 1900 to 1913, much of its stone walls and remaining gates and yagura were destroyed with construction work on the expansion of Uwajima Port.