Airflow Use Cases
Where Airflow fits
Airflow is a batch orchestration platform: it schedules and coordinates work, but it does not process data itself. Its sweet spot is multi-step pipelines with dependencies, retries, and observability - ETL/ELT loads, nightly model training, report generation, data quality checks, and cross-system synchronization. A DAG expresses what runs, in what order, and when; the actual compute happens in operators that call external systems (a database, a Spark cluster, an API, a warehouse).
Where it does not fit
Questions love to test boundaries. Airflow is not a streaming engine: it schedules on intervals, not per-event millisecond latency, so real-time event processing belongs to streaming tools. It is not a data storage or transformation engine - heavy in-warehouse SQL is usually pushed down to the warehouse, with Airflow triggering and checking it. It is not ideal for very long-running single tasks that exceed scheduler patience, nor for dynamic per-user interactive workloads. Also, DAGs are meant to be mostly static: the scheduler parses them repeatedly, so generating thousands of tasks at parse time from a database query is an anti-pattern.
Worked example
A team needs a nightly job that extracts orders, loads them to a warehouse, runs a transformation, and then validates row counts. Airflow fits perfectly: four dependent tasks, a schedule, retries, and alerting on failure. If the same team asked for sub-second reaction to each new order, Airflow would be the wrong tool.
Traps to watch
Don't confuse "orchestrates a Spark job" with "is a Spark replacement." Don't assume Airflow guarantees exactly-once execution; tasks can retry, so downstream logic should be idempotent.