Born in Kensington, London, he was educated at Stowe School, Buckinghamshire, between 1923 and 1927 (under headteacher, John Fergusson Roxburgh), and at Trinity College, Cambridge, where he graduated with an Ordinary BA (a ‘pass degree’).

After the Second World War, Oliver resumed his legal career and having served in the Army, he was appointed as a Judge Advocate in 1952, which saw him try cases around Europe.

He became the Brooklands outer lap record holder in his 8-litre special Barnato-Hassan Bentley racing car on 5 August 1935 with a time of 69.85 seconds, attaining an average speed of 142.60 miles per hour (229.49 km/h).

However this record stood for only 2 months 2 days, as on 7 October John Cobb regained the title in his Napier Railton with a speed of 143.44 miles per hour (230.84 km/h).

In later life, he brought a farm in Leicestershire, where he lived with his second wife, Jane (b 1933) who he married in 1974 in Barnstaple, Devon.