As one of the senior statesmen (Genrō) in Japan during that period, he had a tremendous influence on the selection of the nation's leaders and formation of its policies.

He was a close boyhood friend of Itō Hirobumi who later became Japan's first prime minister, and he played an active part in the sonnō jōi movement.

Desiring to rid Japan of foreigners, he and Takasugi Shinsaku set fire to the British legation in Edo in January 1863.

In 1876, Inoue was asked to assist in the field of foreign affairs, and was involved in the conclusion of the Japan-Korea Treaty of 1876 as vice-ambassador extraordinary and plenipotentiary.

However, Inoue came under public criticism for his failure to negotiate a revision of the unequal treaties, his building of the Rokumeikan, and support of its Westernizing influences, which forced him to resign in August 1887.