He grew up in Hannibal, approximately 100 miles (160 km) northwest of St. Louis, after his father, a railroad engineer, was transferred there.

Phillips was an all-around athlete at Hannibal High School, where he played baseball, basketball and football, in addition to competing in track.

He was promoted to the Class-C St. Joseph Cardinals in 1951, ending with a .307 average and 28 stolen bases while leading Western Association hitters with 14 triples.

The Cards already had Stan Musial, Red Schoendienst, Ray Jablonski, Enos Slaughter and Rip Repulski in the outfield.

Phillips stuck it out in the minors until 1960, playing for eight different teams in the Cardinals, White Sox, Orioles, Braves and Phillies farm systems.