When Ujisato’s son, Gamō Hideyuki was moved to Utsunomiya in 1597, the castle was turned over to Uesugi Kagekatsu, as part of their vast 1,200,000 koku holdings based in Aizu.

However, after the Battle of Sekigahara, the Uesugi were stripped of most of their holdings by Tokugawa Ieyasu, and were reduced to a 300,000 koku domain centered on Yonezawa.

However, due to the reduction in size of their holdings as compared with Aizu and the suspicion with which the clan was viewed by the Tokugawa shogunate, the defensive earthen works were not faced with stone, and donjon was kept to a modest three-stories.

In 1664, the clan revenues were halved to 150,000 koku, forcing it to lay off many samurai, and for many others to take up part-time farming.

All remaining structures were demolished in 1873 and in the second bailey, the present-day Yonezawa city hall was constructed.