Apache Airflow is an open-source workflow management platform for data engineering pipelines.

It started at Airbnb in October 2014[2] as a solution to manage the company's increasingly complex workflows.

Tasks and dependencies are defined in Python and then Airflow manages the scheduling and execution.

DAGs can be run either on a defined schedule (e.g. hourly or daily) or based on external event triggers (e.g. a file appearing in Hive[5]).

[6] Three notable providers offer ancillary services around the core open source project.