Kelley grew up in Fullerton, California, where he was an outstanding athlete at Sunny Hills High School.
He attended California Lutheran University, a National Association of Intercollegiate Athletics (NAIA) member and played college football.
For college, Kelley played linebacker for Cal Lutheran Kingsmen in Thousand Oaks, CA, a team he helped to win the NAIA National Championship in his junior year in 1971.
He was selected by the Associated Press as a first-team linebacker on the 1972 Little All-America college football team.
He finished his college career with seventeen interceptions, which was a school record, and as a punter averaging 34.6 yards per punt.