The first Kakegawa Castle was built by Asahina Yasuhiro in the Bunmei era (1469–1487), a retainer of the warlord Imagawa Yoshitada to consolidate his holdings over Tōtōmi Province.

Yamauchi Kazutoyo completely rebuilt the castle per the latest contemporary designs, and the current layout and much of the stone walls and moats date from his period.

The castle was kept in repair through the Bakumatsu period, however it suffered from extensive damage in 1854, due to the Ansei Tōkai earthquake.

Kakegawa Castle remained in ruins through the Shōwa period, with the exception of the Ni-no-Maru Goten (二の丸御殿) (daimyō's mansion), built by Ōta Sukekatsu after the earthquake, and registered with the government in 1980 as an Important Cultural Property.

[2] In April 1994, sections of the innermost bailey (honmaru), including some walls, a yagura, and the keep (tenshukaku), were reconstructed using the original methods.