Airflow Architecture Components
The moving parts
Modern Airflow separates concerns across several processes sharing one metadata database (Postgres or MySQL):
- Scheduler - parses DAG files, creates DAG runs and task instances, and queues tasks whose dependencies are met.
- DAG processor - in newer versions, a separate process that parses DAG files and writes them to the database, lightening the scheduler.
- Executor - decides where tasks run:
LocalExecutor(processes on the scheduler host),CeleryExecutor(workers via a queue such as Redis/RabbitMQ),KubernetesExecutor(one pod per task). - Workers - execute task code (Celery/Kubernetes setups).
- Webserver - serves the UI and API by reading the metadata database.
- Triggerer - runs deferrable operators' async triggers so workers are not held.
- Metadata database - the source of truth for all state.
Traps to watch
A classic trap is thinking the webserver executes tasks - it only reads state. Another is assuming the scheduler runs task code directly; it queues work for the executor. Also, logs typically live in configured remote storage or local files, not in the metadata database. And in a Celery setup, the queue broker is not the metadata database; they are separate services.
Rule of thumb
Ask "who owns this responsibility?" Scheduler decides, executor places, worker runs, webserver displays, database remembers, triggerer defers.