[4] Whilhoite enrolled at the University of Southern California and, while there, played at the guard position on the USC football team in 1951 and 1952.

Wilhoite set up the game-winning touchdown when he intercepted a Paul Cameron pass and returned it 72 yards to UCLA's eight-yard line.

[5] The Trojans won the 1953 Rose Bowl by a 7–0 score over Wisconsin, and Wilhoite was a consensus selection for the 1952 College Football All-America Team.

[6] Wilhoite was selected by the Cleveland Browns in the 12th round of the 1953 NFL draft, but he instead pursued a career as a boxer.

In September 1953, he won his second professional bout by a second-round technical knockout (TKO) over Clayton Mann in a match at the Olympic Auditorium in Los Angeles.