Handley played baseball for Soldan High School and the Jerome Goldman Post American Legion team in St. Louis, Missouri.

An article in the May 25, 1937, issue of the Pittsburgh Post-Gazette described him as the "leading quarterback of the conference, featured schedule with his passing running and kicking.

"[2] In 1933, Handley won the Most Valuable Player Award in football in the Interstate Intercollegiate Athletic Conference, and he was an all-conference second baseman two years.

[1] In 1939 Handley hit a career-high average of .285 and tied for the National League lead in stolen bases (17), despite suffering a serious beaning that kept him out of the lineup for 52 games.

Jackie Robinson named Handley, who played for the Phillies in 1947, as the first opposing player to wish him well,[3] and stated that he even apologized for the behavior of his teammates, who were acting on instructions of Ben Chapman, their manager, who was racist.