It is the first major rise in terrain along the eastern coastline of Port Phillip, between Melbourne and the Mornington Peninsula.

It was named after local Frankston fisherman, James Oliver, who built the first cottage atop the hill in the mid-19th century, from where he kept watch for fish in the waters below.

Real estate agents in Melbourne generally refer to any property along the Nepean Highway strip from the Frankston Waterfront in the north, to Humphries Road in the South, as being on Olivers Hill.

It can also be reached by bus services 781, 784 and 785,[5][6][7] departing from the Young Street public transport terminus in the Frankston CBD.

Olivers Hill is also well known among Melbourne cyclists as being the halfway or turn around point for the 85 km ride from St Kilda along Beach Road, a famous Australian cycling route.