Prior to undertaking his artistic career, he served in paramilitary and military groups such as the Special Night Squads (SNS), the Haganah, and the British Army during World War II.
Rilov became disenchanted with Zionism after a series of experiences that culminated in his refusal to obey an order to shoot an elderly man and his expulsion from the Haganah.
Later, Rilov developed a close friendship with the Israeli poet, Hana Senesh, who had been a student at the Agricultural College for Girls in Nahalal.
However, he became disillusioned after a Haganah commander ordered him to shoot an elderly peasant from the village of Ma'alul, who had returned to the lands he once cultivated that had been sold to Nahalal by absentee Arab owners.
Shunned by his family and friends after he was court-martialled and expelled from the Haganah for disobeying orders, Rilov left Nahalal for Tel Aviv where he joined the Palestine Communist Party.