After widespread peasant revolts occurred in Morioka Domain in 1836 and 1847, largely in protest to economic policies and food shortages, Toshitada went into voluntary retirement, and Toshitomo nominally became the new daimyō.

After around a year, sentiment against Toshitada had died down somewhat in Morioka, so Toshitomo was forced into retirement, and replaced by his more pliable younger brother Nanbu Toshihisa.

Toshitomo was very angry with this situation, especially as the policies which had resulted in widespread revolts were continued, and the samurai of the domain split into factions supporting either the son or the father.

Relations between the two were so bad that at one point a retainer of Toshitada attempted to kill Toshitomo by poisoning.

Toshitomo remained in Edo through the remainder of the Bakumatsu period, and only visited Morioka as a private citizen after the Meiji Restoration.