In 1863, he enrolled at the University of St Petersburg, where he studied Oriental Languages and graduated with the degree of master of history in 1868.

From 1864 Harkavy was secretary of the Society for the Promotion of Culture Among the Jews of Russia, and from 1873 he was one of the directors of the Jewish community of St. Petersburg.

In 1876 he was appointed head of the Oriental Division in the Imperial Public Library, an astonishing achievement for a Jew under Czarist society's anti-Semitic policies.

Among his theories, he speculated that certain groups of Eastern European Jews, such as the Krymchaks, Karaims and even many Ashkenazim, might be descended from the Khazars.

This theory, largely debunked by modern genetic testing, inspired Arthur Koestler's The Thirteenth Tribe, which took Harkavy's hypothesis to an extreme.

Harkavy's aim here was to prove that the first Jews who settled in South Russia did not come from Germany, as was supposed by Grätz and other historians, but from Greece through the Black Sea region and the Crimea, and from the Orient by way of the Caucasus.

Harkavy contributed many valuable articles on the early history of the Jews in Russia to Meassef Niddaḥim (supplement to Ha-Meliẓ, parts i. and ii.

), Russko-Yevreiski Arkhiv (1883), Brüll's "Jahrbücher (1876), Voskhod (1881–84), Ben 'Ammi (part i., St. Petersburg, 1887), Ḥadashim gam Yeshanim (in Ha-Miẓpah, vol.