At the end of the Edo period, Yoshida Castle was home to the Inaba clan, daimyō of Tateyama Domain.

Due to its strategic location on a river crossing, the castle was involved in numerous battles during the Sengoku period, changing hands several times, and was destroyed and rebuilt on several occasions.

Following the Battle of Odawara in 1590, Toyotomi Hideyoshi ordered the Tokugawa clan to relocate to the Kantō region and assigned the castle to Ikeda Terumasa.

The final daimyō of Yoshida, Matsudaira Nobuhisa, surrendered the castle without resistance to the Meiji government in 1868.

Much of the castle grounds remained in the hands of the military (as the garrison of the IJA 18th Infantry Regiment) until the end of World War II.