Raye and College Football Hall of Famers Bubba Smith (from Texas), George Webster (South Carolina) and Gene Washington (Texas) arrived at Michigan State from the segregated South as part of head coach Duffy Daugherty's Underground Railroad.

Raye was drafted by the Los Angeles Rams for the position of cornerback but was quickly traded to the Philadelphia Eagles.

Raye began his coaching career in 1971 at his alma mater, Michigan State, where he stayed for five years (1971–75).

He coached in the NFL for a total of 36 years with 10 different teams, serving as offensive coordinator for 13 seasons: 1983–84 with the Los Angeles Rams, 1985–86 with the Tampa Bay Buccaneers, 1990 with the New England Patriots, 1998–2000 with the Kansas City Chiefs, 2001 with the Washington Redskins, 2004–05 with the Oakland Raiders (where he was also assistant head coach), and 2009–10 with the San Francisco 49ers.

[4] His son, Jimmy Raye III, is currently the senior executive advisor to the GM for the Cleveland Browns.