He immigrated to the United States as a pre-teen leaving his family in Lemnos and settling in New Jersey in 1920.
He received his bachelor's degree from Rutgers and his doctorate from the University of Wisconsin–Madison, where he studied under Alexander Vasiliev.
From 1936 to 1938, he participated in Grégoire's seminar where he met his future wife Madeleine Schiltz and befriended the likes of Nicholas Adontz and Paul Wittek.
That interest influenced both him and his studies, notably The Armenians in the Byzantine Empire (Byzantinoslavica, 1961) and A Note on the Ethnic Origin of Emperor Maurice (Byzantion, 1965).
Charanis also spent some time at the Aristotle University of Thessaloniki in Greece, and upon his return to the United States joined the Rutgers faculty in 1938, becoming Voorhees Professor of History in 1963.