He was an excellent all-around athlete and led his high school baseball team to a championship while averaging 14 strikeouts a game.

Not long after expressing his concern over playing with Hogan in the 1951 Colonial National Invitation Tournament, which ended badly for him,[6] he faced him in the final group at the 1952 U.S. Open.

It turned out to be one of Oliver's greatest performances as he came from five behind golf's leading man over the final 36 holes in the Texas heat.

[7] In 1940 Oliver finished in a tie with Lawson Little and Gene Sarazen at the 1940 U.S. Open, but in a highly controversial decision was disqualified for teeing off 30 minutes early over weather concerns (under current rules, tournament directors reserve the rule to advance round start times, group players in three, and using both the first and tenth tees in case of approaching weather).

[9] He was the medalist in the stroke play qualifier of the PGA Championship in 1954, but lost in the third round to eventual champion Chick Harbert.

In the 1953 matches at Wentworth, England, he teamed with his boyhood friend Dave Douglas (the only other golfer from Delaware to win on the PGA Tour) to defeat Peter Allis and Harry Weetman.

[12] To spend more time with his family and gain a regular paycheck, he held head professional positions at Hornell, New York, Kenmore, Washington, and at Blue Hill Country Club in Canton, Massachusetts.

Two weeks after finishing ninth in the 1960 Houston Open, Oliver was diagnosed with cancer and had part of a lung removed in late May in Denver.

Presidents Kennedy and Eisenhower, Bob Hope, Bing Crosby, Ed Sullivan and many more joined a national committee to raise funds for his family and to fight cancer.

[16] During the 2022 BMW Championship in Wilmington, Delaware Oliver was inducted by the Western Golf Association into the Caddie Hall of Fame.