[1] A native of East New York, Brooklyn, Schwartz was born to Hungarian mother and a Polish American father, who worked in textile manufacturing.
In 1965, Schwartz moved to the Netherlands to research Dutch Golden Age painting, never to return to the United States and abandoned completing his doctoral degree.
In 1966, Schwartz took up various jobs, including as a translator of Dutch and German texts, an assistant to Jan Gerrit van Gelder at Utrecht University, and editor of the journal Simiolus.
Two years later, Schwartz also worked under Horst Gerson editing publications, and married Loekie Hendriks.
[citation needed] Schwartz continued research on Rembrandt, publishing several more books over the years and establishing himself as an expert on the artist.