Rabbi Zilber married Gita Zeidman, and they had four children – Sarah, Ben Tzion Chaim, Chava, and Fruma Malka.
After World War II, Rabbi Zilber was imprisoned in a gulag for a technical accidental crime (a fellow Jew left some illegal papers by him, and a search revealed them.
When he received an invitation for an interview in the KGB, Yitzchok Zilber fled from Kazan, and after a long journey stayed in Tashkent and was able to move his family there a short time later.
When he arrived in Israel, Rabbi Zilber was shocked to find that the vast majority of Russian-speaking Jews were not observant and for the most part, completely ignorant, of Jewish law and tradition.
In 2000, he established the Toldos Yeshurun organization to provide Jewish education to the secular Russian Jews, which continues his work today, under the guidance of his only son, Rabbi Ben Tzion Zilber.
Rabbi Yitzchok Zilber died in 2004, on the eve of Tisha b'Av (the Ninth of Av), the greatest day of tragedy in Jewish history.