Following his service in the military, Silva attended San Jose State University, graduating with a bachelor's degree in education in March 1950.
Silva became first athletic director at San Jose City College and he coached football, basketball, baseball, and track teams during his tenure at the school.
He led the football team to four championships during the 1950s and coached notable players that eventually played in the NFL such as Oakland Raiders quarterback Chon Gallegos and Chicago Bears offensive lineman Jim Cadile.
In the Pac-8, Silva officiated basketball games involving Lew Alcindor (better known as Kareem Abdul-Jabbar), who played for the University of California, Los Angeles (UCLA) Bruins from 1965 to 1969.
The highlight of Silva's career was being selected as referee for Super Bowl XIV between the Los Angeles Rams and Pittsburgh Steelers in 1980.
A lifelong friend of Silva said, "Fred had a great sense of humor and a razor-sharp wit, which occasionally served him well in tense situations during NFL games.