[4] As a member of the US National Track & Field team, Frazier participated both in the Olympic and Pan-American games.

He would later become a full-time administrator and remained with the university for a total of 23 years, eventually becoming the Senior Associate Athletics Director.

[11][12] Frazier was fired the day after he failed to re-sign football coach June Jones.

[13][14] In 2008 Frazier was named associate athletic director for sports administration at Temple University, and stayed there until 2011.

[20] He was named as one of The 50 Most Powerful African Americans in Sports in the March 2005 issue of Black Enterprise magazine.

On March 3, 2007, he was loudly booed by the crowd following the final home game of the University of Hawaii's head basketball coach Riley Wallace.

[27][28] Frazier is widely viewed as having forced Wallace to resign by including a "no-extension" clause in his final contract.

"[29] Frazier was also criticized by both the media and public for ongoing delays and a perceived lack of integrity in finalizing the full 2007 Hawaii Warriors football team schedule.

Frazier admitted that he may have miscalculated in regards to the schedule but stands by his record at Hawaii citing, in part, his balancing of the athletics budget.

[11][12] On April 6, 2009, Jim Bolla—who Frazier hired in 2004—was fired as the women's basketball coach for Hawaii after former players complained of a "pattern of verbal abuse" from Bolla.