He began his 18-year professional career in the minor leagues in 1932, and after a one-inning trial with the Cardinals in April 1936, played his first full MLB season in 1938 as a member of the Athletics.

He led the American League in earned runs allowed with (144) in 1939, and overall won only 20 of 57 decisions in his first of two stints in Philadelphia, playing for a team that lost an average of 96 games a year between 1938 and 1941.

During the latter year, on June 30, Potter's contract was sold to the Red Sox, who used him in only ten games and 20 innings pitched before assigning him outright to their top farm club, the Louisville Colonels.

With World War II depleting major league rosters of playing talent, Potter proceeded to post three consecutive stellar seasons for the 1943–1944–1945 Browns, winning a total of 44 games with earned run averages below 3.00.

In 1948, he was reacquired by the Athletics in May, but on June 13, after less than a month with Philadelphia, he was "fired" by manager Connie Mack after a losing effort in relief against his old Brownie teammates.

However, future Baseball Hall of Famer Warren Spahn, who relieved Potter, shut the door on the Indians on only one hit, and allowed Boston to come back to win, 11–5.

In World Series play, he posted a record of 0–1 and an ERA of 3.60, allowing six earned runs, 16 hits and five bases on balls, with seven strikeouts, in 15 innings pitched over four different appearances.