Born in Syracuse, New York, Rafferty attended Fayetteville-Manlius High School in Manlius, where he received all-league honors as a defensive lineman in football and also in lacrosse.
He accepted a scholarship from Penn State University, where he played for head coach Joe Paterno and was converted into an offensive lineman.
The next year, he was named the starter at right guard, performing also as a long snapper for field goals and extra points, on a team that won Super Bowl XII over the Denver Broncos.
In 1981, he was moved to center after Robert Shaw went down with a career-ending knee injury, becoming the anchor of an offensive line that would enable the Cowboys to reach two NFC Championship Games.
[4] Among his memorable moments, he and Scott teamed on the block that cleared the way for Tony Dorsett's 99-yard run against the Minnesota Vikings on Monday Night Football in 1983.