The main branch of the Inoue clan was transferred numerous times throughout the Edo period.

The final daimyō of the main branch, Inoue Masanao (1837–1904), was made a viscount (shishaku) under the kazoku peerage system.

A minor 10,000 koku domain, it remained in the hands of the Inoue clan until the Meiji Restoration.

Takaoka Domain (10,000 koku) remained in the Inoue clan until the Meiji Restoration.

Its final daimyō, Inoue Masayori (1854–1904) served in the early police forces of the Meiji government and was made a viscount.