He lived and worked on the family's small cotton farm, until his mother moved with her youngest children to Benton Harbor, MIchigan after the death of Walker's sister and to escape his abusive father.

[1][2] Walker played high school basketball for the Benton Harbor High School boys basketball team, and earned a scholarship to Bradley University, where Walker was a two-time consensus All-America in 1961 and 1962, averaging 24.4 points per game and 12.8 rebounds over three years.

[8][12] That Alex Hannum-coached team, which also featured center Wilt Chamberlain, guards Hal Greer and Wali Jones, and sixth man Billy Cunningham, ended the eight-year championship run of the Boston Celtics.

[13][8] Walker played his final six seasons with the Chicago Bulls, and never averaged less than 19.2 points and 5.0 rebounds a game.

On February 6, 1972, Walker scored a career-high and then-team-record 56 points during a Bulls win over the Cincinnati Royals.

He is the author of a memoir entitled Long Time Coming: A Black Athlete's Coming-of-Age in America (1995).

[17] He was executive producer of the 1979 television miniseries, Freedom Road, that starred Muhammad Ali and Kris Kristofferson.

[18] On February 24, 2012 (two days after his 72nd birthday), it was announced that Walker was elected to the Naismith Memorial Basketball Hall of Fame by the veterans committee.