According to Dr. Tomoteo S. Oracion of Silliman University, during the pre-colonial periods, a certain locality is named after the most popular person living in it.

[6] During the Spanish regime, the original town plaza was located at the beach, which was transferred further inland due to a massive flood.

Old municipal profiles refer to Ayungon as Todos los Santos, though there are no legends to explain that Hispanic name, just as there are no tales elaborating on the ruins of apparently Hispanic fortifications on the Tampocon II shoreline, perhaps because Ayungon's colonial past was not entirely its own.

During World War II, numerous houses were burned, as well as people being tortured and killed.

Maximo Enardecido, the mayor at the time, evacuated the municipal government to Pangi, a mountainous village located south of the Poblacion.