In this event he finished in fourth, fifth, second and eighth place at the 1964, 1968, 1972 and 1976 Summer Olympics, respectively, and won a bronze medal at the 1975 Pan American Games.

During his career, Silvester won five AAU discus titles and set four world records, two in 1961 (60.56 m and 60.72 m) and two in 1968 (66.54 m and 68.40 m).

His personal bests were 70.38 m in the discus (1971, unofficial world record) and 20.01 m in the shot put.

After retirement, he worked as professor of physical education at Brigham Young University.

Silvester won gold medals for the M45 Discus Throw at the 1983 and 1984 Masters National Outdoor Championships.