[4][5] Castor received a football scholarship for Duke University where he established the single-season record for average yards per reception (20.70) in 1982,[3][6] He also caught 46 passes for 952 yards and 13 touchdowns during his senior year at Duke, becoming the ACC Player of the Year and a second-team All-America selection by the Associated Press.

He earned All-American status by running the 400 m leg of their 4th-place distance medley relay team at the 1981 NCAA Indoor Track and Field Championships.

[7] In the 1983 USFL Draft, Castor was a fifth round selection of Tampa Bay Bandits.

[14] In 1994, Castor was inducted into the Cary High School Imps' Hall of Fame for football and track.

[16] In 2014, the Atlantic Coast Conference named Castor to the All-Time ACC Football Legends.