His pro career began at the age of 18[1] after he graduated from Birdsboro High School,[2] when he was signed by the Philadelphia Athletics' legendary manager, Connie Mack, in 1936.

He toiled in the Yankees' farm system through 1942, then performed World War II service as a member of the United States Coast Guard (1943–1945).

[1] Hampered by a sore elbow,[2] Gumpert appeared in fewer innings the following season and his ERA ballooned to 5.43, but still compiled a 4–1 record, as the Yankees won the American League pennant.

He was exclusively a relief pitcher for the Yankees in the early months of 1948, posting a 2.88 ERA in 15 games and 25 innings pitched, but his contract was sold to the second-division White Sox on July 25.

On May 1, 1951, Gumpert became part of baseball history when he allowed Mickey Mantle's first home run,[3] during a relief appearance at Comiskey Park.

His value improved by his All-Star selection, Gumpert was traded that offseason, on November 13, 1951, along with outfielder Don Lenhardt to the Boston Red Sox for infielder Mel Hoderlein and pitcher Chuck Stobbs.

[1] After getting into ten games for the 1952 Red Sox, with one start, he was traded again on June 10, this time to the Washington Senators with fellow hurler Walt Masterson for Sid Hudson, another right-hander.