As a narrow area where the Tōkaidō highway connecting Kyoto with Edo must pass between the Pacific Ocean and the foothills of Mount Fuji, Numazu commanded a strategic location and was also a good port for shipping to the Izu Peninsula.

The original castle consisted of three concentric enclosures, each surrounded by a moat connected to the Kano River, with Umadashi-style gates.

Following the Battle of Sekigahara, Tokugawa Ieyasu recovered his ancestral possessions in Suruga, and gave Numazu to his veteran general, Ōkubo Tadasuke in 1601.

In April 1777, when the former wakadoshiyori Mizuno Tadatomo was transferred from Ohama Domain in Mikawa province to Numazu, he was assigned revenues of 20,000 koku and authorized to build a castle.

As this was in the middle of the period of "Great Peace" under the Tokugawa shogunate, the new Numazu Castle used only a portion of the former site, and did not make use of the former third bailey.