[1] The first castle on this site dates to the early Muromachi period, when the area was under the control of the Tendō clan.

The foundations of the present castle were built by Takenaga Yoshitada, a retainer of the Mogami, in 1535.

Thereafter the castle and Kaminoyama Domain passed through a number of daimyō clans, often for only a generation or two, and its revenues were often reduced.

In 1697, the castle came under the control of the Fujii branch of the Matsudaira clan, which continued to rule over Kaminoyama Domain until the Meiji restoration.

In 1982, on the site of the second bailey, a reconstruction of a “typical” Edo period castle tower was constructed out of concrete to serve as a tourist attraction and to contain a local history museum.