The town's name originates from the nearby coastal feature of the same name, a series of rocky ledges on the point that was first described in an 1875 hydrographic survey.

In 1952 there were three squatters' shacks that had been built in the reserve and once a road was completed into the area in 1953 more people began to request land leases.

In 1963, divers discovered the wreck of the Dutch East India Company ship Vergulde Draeck (Gilt Dragon) which sank on 28 April 1656 after striking reef near Ledge Point.

Damage by looters had left little intact, but over several months a quantity of artefacts was recovered, which are now displayed at maritime museums in Geraldton and Fremantle.

The area is also known for the 1983 wreck of the Jackup oil drilling rig Key Biscayne, which toppled over in storms about 10 nautical miles (19 km) offshore whilst under tow.