Taoka, an orphan, grew up as a young street fighter in Kobe under the guidance of Noboru Yamaguchi, the Yamaguchi-gumi's boss.

Consequently, severe police crackdowns and arrests would be constantly imminent, making it a high cost venture for relatively low profit.

In his time as boss, Taoka expanded the Yamaguchi-gumi from a small strikebreaking gang on the Kobe docks to the world's largest criminal syndicate, with over 10,000 members during its peak.

He had survived a 1978 assassination attempt when he was shot in the back of the neck by a member of the Matsuda-gumi (松田組), a rival gang to the Yamaguchi-gumi, during a limbo dance exhibition at a nightclub in Kyoto.

His son Mitsuru Taoka (1943–2012) was a producer of films in Japan and was married to actress Eiko Nakamura until her suicide at their home by gas poisoning in 1975.