During the Edo Period, it was the center of Marugame Domain, ruled by the tozama Kyōgoku clan under the Tokugawa Shogunate.

During the Muromachi period, this area was ruled by the Kagawa clan, originally from Sagami Province, from their stronghold at Amagiri Castle in what is now the city of Zentsuji.

In 1640, the Ikoma clan was demoted to Yashima Domain in Dewa Province by the shogunate in 1640 due to attainder following an uprising, and their holdings were divided.

After the Meiji restoration, in 1869, a fire swept through the castle and destroyed the daimyō palace and the Inui Yagura turret.

However, the auction was cancelled by order of the Imperial Japanese Army, which laid claim to the castle site for use by the newly formed IJA 12th Infantry Regiment.

This area was opened to the public in 1919 as "Kameyama Park", and in 1933, the Enjukan, a villa owned by the Kyōgoku clan in the castle town was relocated to its current site within the San-no-maru Bailey.