Inoue was born into a samurai family in Higo Province (present-day Kumamoto Prefecture), as the third son of Karō Iida Gongobei.

He became a protégé of Ōkubo Toshimichi, and accompanied him to Beijing (China) for negotiations with the Qing court following the Taiwan Expedition of 1874.

After Ōkubo's assassination, he worked closely with Itō Hirobumi and Iwakura Tomomi, and became a member of the Genrōin.

In 1875, based upon his experiences in Europe, Inoue published two volumes of documents called Ōkoku Kenkoku Hō (憲法意見控) ("Constitutions for Kingdoms"), which was primarily a translation of the Prussian and Belgian constitutions with Inoue's own commentary, which he submitted to Iwakura Tomomi.

Iwakura recognized Inoue's talent and assigned him to work on the project for drafting a new constitution for Japan.