McGhee graduated from New England Conservatory in 1949 and worked for a short time with trumpeter Roy Eldridge and local Boston musician Fat Man Robinson.
From 1957–1963 he worked in Lionel Hampton's band, touring the United States, Europe, and the Far East.
Among his students were saxophonists Bill Pierce, Javon Jackson, Donald Harrison, Antonio Hart, Sam Newsome, Richie Cole, Greg Osby, and Ralph Moore.
The band included Bob Wilber, clarinet, Ernie Wilkins, saxophones, Teddy Wilson, piano, Alan Dawson and Terri Lynne Carrington, drums and Hampton on vibraphone.
The Golden Men of Jazz played concerts throughout Europe and on returning to the United States played for President George H. W. Bush in Washington, D.C.[3] In May 2006, McGhee was awarded an Honorary Doctorate of Music from Berklee College of Music.