Jovan Karamata was born in Zagreb on February 1, 1902, into a family descended from merchants based in the city of Zemun, which was then in Austria-Hungary, and now in Serbia.

Being of Aromanian origin, the family traced its roots back to Pyrgoi, Eordaia, West Macedonia (his father Ioannis Karamatas was the president of the "Greek Community of Zemun"); Aromanians mainly lived and still live in the area of modern Greece.

In 1914, he finished most of his primary school in Zemun but because of constant warfare on the borderlands, Karamata's father sent him, together with his brothers and his sister, to Switzerland for their own safety.

He spent the years 1927–1928 in Paris, as a fellow of the Rockefeller Foundation, and in 1928 he became Assistant for Mathematics at the Faculty of Philosophy of Belgrade University.

He was one of the founders of the Mathematical Institute of the Serbian Academy of Sciences and Arts in 1946.

In 1931 he married Emilija Nikolajevic, who gave birth to their two sons and two twin daughters.