After his release he met the socialist Kōtoku Shūsui, who offered him a position at a newspaper he was editing, but Yamakawa declined and moved back to his hometown.

He moved back to Tokyo and started working at the Heimin Shimbun in early 1907, where he met lifelong friends Sakai Toshihiko and Arahata Kanson.

In 1922, as younger converts to Bolshevism were becoming impatient, Yamakawa along with Sakai and Arahata agreed to found an illegal Communist Party.

In it, he criticized the anarchist faction which had been dominant within the socialist and labour movement in Japan for being idle dreamers who failed to obtain anything concrete that benefited the working class.

Yamakawa became the most influential theoretician of the small Communist Party which, while illegal, was popular among left wing students and academics.

In 1927 Yamakawa and others established a loosely organised Marxist group, the Rōnō-ha (Labour-Farmer Faction), which influenced socialist and communist activists through writings and discussions while refraining from open political action.

Yamakawa withdrew from active politics in 1931, but was nevertheless thrown in prison in 1937 when the government was clamping down on dissent after invading China.