The underlying name is the Gaelic "druim mòr" or "big ridge", and it has been suggested that this reflected the motte associated with the castle of the Adairs of Kinhilt, whose lands were granted in 1602 by King James VI.

It was opposed by the feudal landowner, the Earl of Stair, and finally abandoned after the failure of the City of Glasgow Bank in 1882; aspects of the village's street layout still reflect plans for the railway.

Shortly before 1860, at Low Curghie less than a mile up the coast north of Drummore, and not far from an extant standing stone, a gravestone was discovered which appeared to date to the 5th or 6th century, with a weathered Latin inscription in which the name "Ventidius" was legible along with another word which translated as "sub-deacon".

The harbour, facing north and shielded by the Rhins from the prevailing southwesterly wind, was developed with a jetty in the early 19th century to serve a lime manufacturing industry.

Failure of the Drummore Harbour Trust to widen its membership beyond the initial two individual members, or to begin its promised investment programme, caused increasing concern.

Even recently there have been significant casualties, including on Mill Street the bank, the butcher's (now wholesale only, as a result of health legislation), and the Harbour Stores (formerly the second general shop, with an additional speciality in fishing supplies).

The village's facilities still however include the Mariners Coffee Shop, the Queen's Hotel, a post office, a volunteer tourist office and a general shop on Mill Street; a garage for repairs (no fuel); a primary school; a doctor's surgery and pharmacy; the Ship Inn on the shore; a bowling club and children's play park; a number of holiday cottages; and two caravan sites near sandy beaches (one of them associated with another pub).