Schwartz was born in New York's East Village on October 23, 1902, not far from the neighborhood of Jewish Lightweight Champion Benny Leonard.

According to one source which may be embellished, the army brawl that launched his career took place when Izzy was barely eighteen around 1919, and was precipitated when Schwartz, refusing to loan money to a fellow soldier, received an ethnic slur.

[4] Schwartz's first official bout in the Army was a torrid match with Kid Pancho, whom he fought to a fifteen-round draw despite being outweighed by eight pounds.

[4] After victories from Billy Kelly, Blas Rodriguez, Willie Davies, and Benny Hall in late 1927, Schwartz was chosen to meet Newsboy Brown for the flyweight title by the New York State Boxing Commission.

Another source attributed superior speed to Schwartz's victory, noting "he proved himself too fast, shifty, clever, and resourceful for the plodding Brown."

[4][5][6] Continually depicted by the press as an underdog, even in this his greatest victory, the Reading Times described Schwartz as a "frail-looking soldier warrior", and noted he was outweighed four pounds by Newsboy Brown.

Schwartz started the fight favorably, but by the end, "Bushy made use of his weight advantage, and Izzy "tired and fell under the gruelling pace.

[8] Schwartz worked for the army again in World War II, directing athletics at the Shenango Replacement Training Center near Pittsburgh, Pennsylvania, after completing a tour of duty overseas.