Airflow Fundamentals cheat sheet

The Dags view lists every parsed DAG with its schedule, tags, owner, and last run state; paused DAGs show a toggle set to off.

Grid view shows a matrix of runs (rows) by tasks (columns); each cell is a colored square whose color encodes the task instance state.

Graph view renders the task dependency structure; clicking a task opens a panel with logs, rendered templates, XCom, and run duration.

The Calendar view maps one box per scheduled run date, colored by run state, and is the fastest way to spot missing or failed runs.

Gantt view plots task start and end times on a timeline, making queueing delays and long-running tasks visible.

Code view shows the DAG file source read-only; the DAG must be reparsed by the scheduler before UI edits to the file appear.

Admin menu exposes Connections, Variables, Pools, and Plugins; these are stored in the metadata database, not in DAG files.

Browse > Task Instances allows filtering across all DAGs by state, execution date, and task, useful for finding stuck queued tasks.

airflow dags list shows parsed DAGs; airflow dags list-import-errors surfaces Python import failures that keep a DAG out of the UI.

airflow tasks test <dag_id> <task_id> runs one task locally without recording state in the metadata database.

airflow dags trigger <dag_id> -e creates a manual run; airflow dags backfill <dag_id> -s -e creates runs for past intervals.

airflow tasks run <dag_id> <task_id> executes a task and does persist state, unlike tasks test.

airflow tasks states-for-dag-run lists task instance states for a single run; airflow dags state reports the DAG run state.

airflow db init (or airflow db migrate in newer versions) creates or upgrades the metadata database schema.

airflow users create --username --role --email registers a UI user; roles include Admin, User, Op, and Viewer.

airflow connections add and airflow variables set manage those objects from the terminal; airflow config get-value reads configuration.

airflow scheduler, airflow webserver, and airflow celery worker start the corresponding architecture components as long-running processes.

Airflow is a batch orchestrator: it schedules and monitors workflows, it does not process data itself, so tasks call out to other systems.

Good fit: ETL/ELT pipelines, scheduled report generation, ML model training pipelines, and cross-system data movement at fixed intervals.

Good fit: backfills of historical partitions, because runs are keyed by logical date and can be recreated for past intervals.

Poor fit: streaming or event-driven processing with sub-second latency; use a stream processor and let Airflow trigger or monitor it.

Poor fit: long-running always-on services or per-record processing loops; tasks should be bounded and idempotent.

Poor fit: workflows needing very high task-per-second throughput; scheduler and database overhead grow with DAG and task counts.

DAGs-as-code means pipelines live in version control and are reviewed like software, which suits teams already using Git-based CI.

Typical pattern: sensors wait for upstream data, transformation tasks run, then a final task publishes or notifies on completion.

A DAG is defined with DAG(dag_id=..., start_date=..., schedule=...); start_date is the anchor from which the first logical interval is computed.

The first run covers the interval starting at start_date; with schedule='@daily' and start_date of 2024-01-01, the first run's logical date is 2024-01-01 and it executes after 2024-01-02 begins.

schedule accepts a cron expression, a timedelta, a preset such as @daily, or None for manually triggered only; schedule_interval is the older parameter name.

catchup=True (default) creates runs for every missed interval since start_date; catchup=False skips to the latest interval only.

max_active_runs_per_dag limits concurrent DAG runs; max_active_tasks limits concurrent task instances per DAG.

default_args supplies shared values such as owner, retries, retry_delay, and email; task-level arguments override them.

Task dependencies are set with a >> b, a << b, or a.set_downstream(b); the bit-shift operators are the preferred syntax.

Idempotent tasks and deterministic templates make reruns safe; use Jinja templates like {{ ds }} and {{ data_interval_start }} instead of hardcoded dates.

Debug by reading task logs in the UI, running airflow tasks test locally, and checking airflow dags list-import-errors for parse failures.

Intra-DAG dependencies order tasks within one DAG; inter-DAG dependencies order whole DAGs across files.

Operators: ExternalTaskSensor waits for a task in another DAG to reach a state; it needs external_dag_id, external_task_id, and a matching execution date.

TriggerDagRunOperator starts another DAG from a task; it can pass a conf dictionary to the triggered run.

Datasets (now called Assets in Airflow 3) let a producer task declare outlets and consumer DAGs declare a schedule on that dataset, giving data-driven dependencies.

The CLI airflow dags trigger can be scripted, but it is not a dependency mechanism inside the scheduler.

Cross-DAG dependencies should avoid cycles; a cycle between DAGs will cause waits that never resolve.

ExternalTaskSensor execution_delta or execution_date_fn aligns differing schedules between the waiting and target DAGs.

Task groups (TaskGroup) organize tasks visually and for dependency wiring but do not change execution semantics.

A DAG run is one instantiation of a DAG for a logical interval; its identity is (dag_id, logical date), so the same interval cannot run twice concurrently.

Run types are scheduled, manual, backfill, and dataset-triggered; the type is visible in the UI and in the run's metadata.

Run states include queued, running, success, failed; task instance states include none, scheduled, queued, running, success, failed, up_for_retry, skipped, and up_for_reschedule.

The logical date (formerly execution_date) labels the interval start, not the wall-clock time the run executes.

data_interval_start and data_interval_end expose the interval boundaries to templates and task context.

A run is marked success only when all leaf tasks succeed or are skipped per trigger rules; a single failed task with all_success rule fails the run.

Clearing a task instance in the UI or CLI creates new task instances so the run can be retried without recreating the run.

The scheduler creates runs; the executor runs tasks; the metadata database records run and task state.

Scheduler: parses DAG files, creates DAG runs on schedule, and queues task instances whose dependencies are met.

Executor: determines where tasks run; options include SequentialExecutor (SQLite, dev only), LocalExecutor (parallel on one host), and CeleryExecutor or KubernetesExecutor (distributed).

Worker: the process that actually executes task code; Celery workers pull from a broker such as Redis or RabbitMQ.

Webserver: serves the UI and REST API; it reads state from the metadata database and does not schedule work.

Metadata database: PostgreSQL or MySQL in production, SQLite only for local testing; stores DAGs, runs, task instances, variables, and connections.

DAG files live on a filesystem or object store that the scheduler and workers can all read; a missing file on a worker causes import errors.

Triggerer (Airflow 2.2+) runs deferrable operators and sensors asynchronously, freeing worker slots while waiting.

The components communicate through the metadata database and, for Celery, the broker queue; there is no direct scheduler-to-worker RPC.

Operators are task templates; instantiating one inside a DAG creates a task with a task_id that must be unique within the DAG.

PythonOperator calls a Python callable given by python_callable; arguments pass via op_args (positional list) and op_kwargs (keyword dict).

BashOperator runs a shell command in command=; it returns the last line of stdout, which can be pushed to XCom.

EmptyOperator (formerly DummyOperator) does nothing and is used to join or fan out branches.

BranchPythonOperator returns the task_id of the branch to follow; unselected branches are skipped.

TaskFlow API with the @task decorator wraps Python functions as tasks and passes return values between tasks implicitly via XCom.

Every operator accepts retries, retry_delay, depends_on_past, trigger_rule, pool, and queue as task-level arguments.

Heavy logic belongs in the callable, not at DAG parse time; top-level code runs on every scheduler parse.

XCom stores small key-value pairs in the metadata database, keyed by dag_id, task_id, run, and key, with a default key of return_value.

A task pushes with xcom_push(key, value) or by returning a value; it pulls with xcom_pull(task_ids=..., key=...).

In templates, ti.xcom_pull(task_ids='upstream') retrieves a value; the value must be JSON-serializable unless a custom backend is used.

XCom is designed for small metadata such as IDs, paths, and counts, not DataFrames or large files; large payloads bloat the metadata database.

For large data, write to object storage and pass the path via XCom.

XCom is written after a task succeeds, so a failed task's return value is not available downstream.

TaskFlow passes function return values automatically as XCom, with no explicit push or pull call.

Clearing a task instance deletes its XCom entries for that run, so downstream pulls may return None after a clear.

Connections store host, login, password, schema, port, and extra as JSON, referenced by conn_id; credentials live in the metadata database or a secrets backend.

Hooks such as PostgresHook use a conn_id to build a client; operators accept the connection id directly, e.g. postgres_conn_id.

Variables are key-value pairs retrieved with Variable.get('key') or templated as {{ var.value.key }}; they are global, not per-DAG.

Variable.get inside a task is fine; calling it at DAG top level hits the database on every scheduler parse, so use {{ var.value.key }} in templates instead.

Sensors are operators that wait for a condition; they occupy a worker slot while poking unless set to deferrable mode or mode='reschedule'.

poke_interval sets seconds between checks; timeout sets the maximum wait before the task fails.

mode='poke' holds the slot continuously; mode='reschedule' releases the slot between checks, which suits long waits and sparse conditions.

Built-in sensors include FileSensor, ExternalTaskSensor, and S3KeySensor; custom sensors subclass BaseSensorOperator and implement poke().