Caffey and teammates, Ray Nitchke and Dave Robinson, were named one of the top 10 best linebacking trios in the history of the NFL by ESPN.
Born and raised in Texas, Caffey started his football career in Thorndale at the age of ten, when he played on a pee wee league that played a team from Oklahoma in the 'Milk Bowl Championship' and shook hands with American legend, Pro Football Hall of Fame member and Olympic gold medalist Jim Thorpe.
He was recruited to play college basketball by Shelby Metcalfe at Texas A&M but decided to try out for the football team when he got there, earning a scholarship and was a 3-year letterman.
Caffey accepted a football scholarship from Texas A&M University, where he played both sides of the ball and led the Aggies in rushing as a fullback in his junior season of 1961.
On May 5, 1964, Caffey was acquired by the Green Bay Packers in the famous 'Jim Ringo' trade and played under Head Coach Vince Lombardi.
[10] Caffey intercepted nine passes, returning two for touchdowns, most notably one for 52 yards against Johnny Unitas and the Baltimore Colts in the 1966 season opener at Milwaukee.
On January 21, 1970, after Lombardi's departure from Green Bay, Caffey, Elijah Pitts, and Bob Hyland were traded to the Chicago Bears for the second overall pick in the 1970 NFL draft (#2-Mike McCoy).
[14][15] Caffey would start all 14 games as outside linebacker for the Bears, beside Hall of Famer Dick Butkus, in the 1970 season under Head Coach Jim Dooley.
Caffey was set to be traded to the Pittsburgh Steelers who were rebuilding their team and wanted him at linebacker alongside future Hall of Famer Jack Ham, when he decided to announce his retirement.
Some of the pall bearers included former NFL greats, Jerry Kramer, Boyd Dowler, Donny Anderson, Tommy Joe Crutcher, Forrest Gregg, and Don and Diron Talbert.