Airflow Fundamentals key terms

A collection of tasks with defined execution order and no loops, written as a Python file. It describes what to run and when, not the data itself.

A single unit of work inside a DAG, usually created by instantiating an operator. Each task has a unique task_id within its DAG.

A reusable template class that defines what a task actually does, such as running Python code, executing a shell command, or waiting on a condition. Instantiating it creates a task.

An operator that runs a Python callable as a task. Arguments are passed through op_kwargs, and the callable's return value can be pushed to XCom automatically.

One specific execution attempt of a task for a particular DAG run and logical date. It carries its own state such as queued, running, success, failed, or skipped.

A single execution of a whole DAG, identified by a logical date or run id. It groups the task instances created for that execution and records overall status.

The timestamp that labels a DAG run and represents the start of the data interval it covers. Previously called execution_date, it is not the wall-clock time the run started.

The time window of source data a scheduled DAG run is responsible for, running from the previous logical date up to the current one. Templates can reference its start and end.

The setting that decides when new DAG runs are created, given as a cron expression, timedelta, or timetable. It controls run creation timing, not the duration of tasks.

A five-field string describing recurring times in minutes, hours, days of month, months, and days of week. Airflow interprets it in the timezone configured for the DAG.

A DAG setting that, when true, makes the scheduler create runs for every missed interval since the start date. Setting it false limits scheduling to the most recent interval.

Deliberately creating and running DAG runs for past logical dates, either through the CLI or the UI. Useful for reprocessing historical data after logic changes.

The earliest logical date from which the scheduler may create runs for a DAG or task. Runs are only created for intervals that begin at or after it.

The relationships that order tasks inside a DAG, expressed with the bitshift operators or set_upstream and set_downstream. They form the edges of the graph.

A per-task setting that decides whether the task runs given the states of its upstream tasks, for example all_success, all_done, or one_failed. Defaults to all_success.

The lifecycle status of a task instance, such as scheduled, queued, running, success, failed, up_for_retry, skipped, or upstream_failed. The scheduler and executor update it.

The number of times a failed task instance may be attempted again before being marked failed. Usually paired with retry_delay to space out attempts.

The waiting period before a failed task is retried, given as a timedelta or, in newer versions, a backoff strategy. It prevents hammering a failing external system.

A small key-value store that lets tasks in the same DAG run pass values to each other. Values are stored in the metadata database and should stay small.

The act of writing a value into XCom under a key, either automatically from an operator's return value or explicitly by calling xcom_push or the task instance method.

Reading a value previously pushed by another task, typically by task_id and key, using templates or the task instance method. Returns None when nothing matches.

A special operator that waits until a condition is true before succeeding, polling at a fixed poke_interval. It occupies a worker slot unless using deferrable mode.

How often a sensor rechecks its condition while waiting. Short intervals react faster but consume more worker resources; long intervals reduce load but add latency.

The maximum time a task, often a sensor, may run before being failed by Airflow. It protects the system from tasks that hang indefinitely.

A stored set of credentials and host details for an external system, referenced by a unique conn_id. Admins manage them in the UI, CLI, or environment variables.

A key-value setting stored in Airflow's metadata database and read at runtime, useful for shared configuration. Values can be marked as secret to hide them in the UI.

A low-level interface that wraps a third-party library and uses a connection to talk to an external service. Operators and sensors typically build on hooks.

Placeholder syntax inside operator arguments that Airflow renders at runtime, giving access to variables, params, and XCom values through a template context.

A dictionary of DAG-level or task-level values that can be overridden per run and referenced in templates. They make DAGs configurable without code edits.

The Airflow component that parses DAG files, creates DAG runs and task instances on schedule, and hands ready tasks to the executor while tracking states.

The component that actually runs task instances, deciding where and how they execute. Examples include LocalExecutor, CeleryExecutor, and KubernetesExecutor.

A process or pod that receives tasks from a queue and executes them. Multiple workers let a deployment scale horizontally beyond a single machine.

The relational store holding DAG definitions, run and task states, connections, variables, and XComs. All Airflow components read and write to it.

The component serving the Airflow UI, letting users inspect DAGs, trigger runs, view logs, and edit connections and variables. It reads from the metadata database.

A named channel that routes tasks from the scheduler to workers with matching queue settings. It enables separating workloads across worker pools.

The command-line tool for managing Airflow, with subcommands to list and test DAGs, trigger runs, run backfills, and manage connections and variables.

The main UI screen showing DAG runs as rows and tasks as columns, with colored boxes for task states. It is the fastest way to spot failures across history.

A UI view that draws the DAG's tasks and dependency edges, letting users inspect a single run's structure and open individual task instance details.

The property that rerunning a task or DAG run for the same interval produces the same result without duplicating data. It is essential for safe retries and backfills.