[4] Raftery graduated in 1959 from the now defunct St. Cecilia High School in Kearny, where he starred in basketball and became the all-time leading scorer in state history with 2,193 points, a record he held for nine years.

[7] Raftery played for the La Salle Explorers men's basketball team under coach Donald "Dudey" Moore.

[11] His 154 wins as a coach places him fifth on the all-time list at Seton Hall behind Honey Russell, P. J. Carlesimo, Frank Hill, and Kevin Willard.

Conference tournament champion Raftery said it was during his senior year in college, when La Salle was competing in the NIT, that legendary New York sportscaster Bob Wolff suggested he eventually consider a career in broadcasting.

“It was just one of those things in the back of my head, and I said, ‘This will keep me in the game that I enjoy.’”[14] He began his broadcasting career as co-host of ESPN’s College Hoops Tonight in 1980.

[16][17] During the 2014–15 collegiate basketball season, CBS/Turner Sports partnered Raftery with Jim Nantz and Grant Hill to make up the primary announcing team for the remainder of the regular season following the arrest of Greg Anthony for soliciting a prostitute, all the way through the NCAA men's basketball tournament and the Final Four.

[18] Raftery was also an analyst with ESPN, primarily partnered with Sean McDonough and Jay Bilas and formerly Mike Gorman for Big East games.

[29] Previously they lived in Florham Park, New Jersey, where they raised four children, Billy, Kristi, Kelli and Suzi.

[8] In 2015, Billy and his father's broadcasting partner, Grant Hill, produced With A Kiss, a documentary about Raftery's first shot at calling the Final Four at age 73.