After graduating from Dwight Morrow High School in Englewood, he was selected by the Atlanta Hawks in the 1975 NBA draft as the first pick in the second round (19th overall), bypassing college for a chance to play professionally.

While fellow notably young draftees Moses Malone (drafted into the American Basketball Association out of high school in 1974, prior to the 1976 ABA–NBA merger) and Darryl Dawkins enjoyed more successful professional careers, Willoughby had a career that was less distinguished.

[2] After that season, Willoughby gained some fame in the 1981 NBA postseason by becoming one of the few players ever to block Kareem Abdul-Jabbar's "skyhook" at its apex.

[3] Willoughby and the Rockets later advanced to that year's 1981 NBA Finals, where he would play a key role in Houston's Game 2 92–90 win by scoring 14 points off the bench.

Though he later regretted skipping college, Willoughby eventually received his degree in communications from Fairleigh Dickinson University in 2001, at the age of 44.