His widowed mother Rachel, struggling to provide for the family, was forced to place three of his siblings in a Hebrew Orphan Asylum.

When Charley was only five, his mother decided to move the family from the Lower East Side to Harlem, a more ethnically mixed section that still contained many Jews.

Although losing the twelve round points decision at the Commonwealth Sporting Club against Genaro on May 23, 1922, the close fight could have gone either way, and Charlie's manager was impressed with his young boxer's ability to learn.

Rosenberg had picked up pointers on bobbing, ducking, and effectively using his left, from Jewish boxing great Benny Valgar, while training at his gym.

He won a twelve-round decision against the up-and-coming local bantamweight Butts in a semi-final bout at New York's Commonwealth Sporting Club on July 8, 1922.

[5] On October 30, 1923, he defeated the prolific Black boxer Danny Edwards, in a ten-round points decision at New York's Pioneer Sporting Club.

Edwards would fight top talent in his long career, including several champions, but he had a three-inch height disadvantage against Rosenberg.

Rosenberg and "Cannonball" Eddie Martin, 1925 Bantamweight Champion of the World, met three times, twice in six-round decisions and once in a draw.

On November 29, 1923, and January 28, 1924, Martin defeated Rosenberg, in close decisions on points, both times in New York's Madison Square Garden.

In their third meeting, a fast and furious affair on April 29, 1924, Rosenberg gave Martin a closer battle which ended in a ten-round draw.

[13][14]Rosenberg defeated Irish boxer Johnny Curtin, one of his top bantamweight competitors, on October 10, 1924, in an important ten round points decision before a crowd at Madison Square Garden.

Cohen fought the best fighters of his generation, including many champions, but their fight at Harlem's Commonwealth Sporting Club, brought limited publicity, likely because it was an early interracial bout.

Ray Arcel, his skilled handler, had prepared him well for the bout, and though Rosenberg had lost thirty-seven pounds in only two months of training, many ringside believed he gained momentum and strength in the final rounds.

[17] The Palm Beach Post noted that Rosenberg won using a "tantalizing left jab and a right uppercut, outboxing Martin at every turn and at the latter part of the match, holding his own in a furious toe-to-toe skirmish.

Rosenberg was described by Time magazine as: wan as if he had spent his life loitering with La Belle Dame Sans Merci beside her autumnal lake, her birdless woods; his face was drawn, his body lean almost to emaciation.

[22] At least one local paper felt Schwartz had fought well against the reigning champion and noted that Rosenberg was well over the bantamweight range and would have a difficult time against Bushy Graham.

[23] Rosenberg lost the World Bantamweight Title to Bushy Graham on February 4, 1927, in a fifteen-round decision at Madison Square Garden.