In the wake of the U.S. entry into World War II, 1942 he enlisted in the United States Merchant Marines Cadet Corps.
In 1947, he joined the United Nations International Refugee Organization in Geneva, serving as reparations director.
In 1949, Schwartz returned to the private practice of law in Washington, D.C. From 1949 to 1962, he was also special legal counsel for the Intergovernmental Committee for European Migration.
During his time as Assistant Secretary, Schwartz encouraged Robert F. Kennedy (now United States Attorney General) to exercise the discretion granted to the Attorney General by the Immigration and Nationality Act of 1952 to allow East European refugees and visitors to come to the United States even if they held suspect political views that would otherwise disqualify them.
In 1967, United States Secretary of Defense Robert McNamara named Schwartz his special assistant for prisoners of war in Vietnam.