During the Muromachi period, the Imagawa clan ruled Suruga Province from their base at Sunpu (modern-day Shizuoka City).

As part of the Tokugawa policy to sap potential rivals of economic strength, daimyōs from around the country were called upon to rebuild Sumpu Castle in 1607 with a triple moat system, keep and palace.

Saris and William Adams met with Ieyasu here to exchange gifts and negotiate terms for the East India Company to trade with Japan.

By 1638, the palace, gates, yagura and other structures were reconstructed, but notably, the donjon was not, since Sunpu was ruled by an appointed administrator, rather than by a daimyō.

His heir, Tokugawa Iesato, was briefly established as daimyō of "Shizuoka Domain" (700,000 koku) in 1868 until its abolition a year later in 1869.

A western-style school, the Shizuhatasha (or Shizuhatanoya) was established in the house which had been built for Clark; and a Canadian missionary, Davidson McDonald, was engaged to run it.

In 1896, a large portion of the inner castle grounds was turned over to the Imperial Japanese Army as a base for the IJA 34th Infantry Regiment.