Smith played college football at North Carolina A&T State University, graduating in 1956, and was three times named a member of the All-Colored Intercollegiate Athletic Association team.

Smith Jr. attended Sterling High School in Greenville, South Carolina, where he played football, basketball, and baseball.

He accepted a football scholarship from North Carolina A&T State University, from which he graduated with a Bachelor of Science degree in physical education and social studies in 1956.

He returned in 1956, but the team had a surplus of running backs (Rick Casares, Bobby Watkins, John Hoffman, J.C. Caroline and Perry Jeter), so he was moved to the defensive side.

In 1956, he was claimed off waivers by the San Francisco 49ers and played the remaining 5 games of the season as a reserve defensive back.

In 1958, while he served as a kickoff returner, he was eased into the running back position until the eleventh game of the season when he ran for 113 yards on seven carries, including and 80-yard touchdown.

In 1959, Hugh McElhenny was moved to flanker and Smith was named the starter at left halfback alongside Joe Perry.