The Japanese 2nd Army was initially raised during the First Sino-Japanese War from September 27, 1894, to May 14, 1895, under the command of General Ōyama Iwao.

It was revived for the Russo-Japanese War from March 6, 1904, to January 2, 1906, under the command of General Oku Yasukata.

The 2nd Army participated in the North China Incident, Tianjin–Pukou Railway Operation, Battle of Xuzhou and Battle of Taierzhuang before being demobilized on December 15, 1938.

The fourth and final incarnation of the Japanese 2nd Army was on July 4, 1942, when it was revived under the command of the Japanese First Area Army in Manchukuo.

Towards the closing stages of the war, on June 30, 1945, it was transferred to the Southern Expeditionary Army Group and was based in the Celebes at the end of World War II.