A native of Columbus, Mississippi, Brewer first came to Ole Miss as a player for the Rebel teams of Johnny Vaught.
In 1974, Brewer was named the tenth head football coach at Southeastern Louisiana and he held that position for six seasons, from 1974 to 1979, compiling a record of 38–24–2.
However, they flopped to a 3–8 record, a severely disappointing mark even considering that they were hobbled by NCAA sanctions for recruiting violations under Sloan.
But the Rebels fell to Tulane in a devastating 14–9 upset on homecoming and also suffered a very heartbreaking loss to a 5–6 Tennessee team at home before rallying to trounce Mississippi State 33–6 in a severe thunderstorm in Jackson.
In his 11 seasons, Brewer led Ole Miss to eight Egg Bowl victories over in-state rival Mississippi State.
However, Brewer's tenure at Ole Miss was marred by allegations of recruiting improprieties that twice led to run-ins with the NCAA.
The Rebels were banned from post-season play and live television for the 1987 season after a two-year investigation found that Ole Miss recruits had received cash and other gifts from boosters.
An NCAA report said that Ole Miss boosters and coaches had offered recruits gifts, including cash and, in one case, a car.
Boosters were also accused of breaking national rules by taking recruits 30 miles outside of Oxford, sometimes to strip clubs in Memphis.
Most damningly, the NCAA alleged that Ole Miss officials knowingly allowed the violations to occur, demonstrating a lack of institutional control of the football program.
Later that year, the NCAA, when announcing severe penalties against the Ole Miss football program, found Brewer guilty of unethical conduct.
In his last several years, Brewer took a more prominent role again in Oxford, appearing on television ads and on a radio show.