It was also called the Cactus Castle (臥牛, Gagyu- jō) Karasuyama Castle was originally built by Nasu Sukeshige in 1418 and was the primary residence of the Nasu clan from 1514 to the end of the Sengoku period.

The castle resisted repeated attacks by the Satake clan and other enemies of the Nasu, but was never taken in battle.

However, in 1590, partly for failing to participate in the 1590 Battle of Odawara, Toyotomi Hideyoshi divided the Nasu holdings, and awarding their ancestral castle temporarily to Oda Nobukatsu, one of the surviving sons of Oda Nobunaga.

With the establishment of the Tokugawa shogunate, Karasuyama Castle became the center of the 20,000 koku Karasuyama Domain, ruled by a succession of daimyo clans (Narita, Matsushita, Hori, and Itakura) before it was awarded in 1725 to a junior branch of the Ōkubo clan.

The castle was abandoned in 1869, and in 1872 the San-no-maru palace collapsed due to weight of heavy snow.