Airflow DAG Authoring key terms

A collection of tasks with defined execution order and no loops, so every task eventually runs after its upstream dependencies finish. Airflow uses it as the top-level unit of scheduling and ownership.

A single node inside a DAG that performs one unit of work, usually by instantiating an operator. Its state moves through queued, running, and terminal values such as success or failed.

A reusable template class that defines what a task actually does, such as run a Python function, execute SQL, or move files. Instantiating it inside a DAG creates a task.

The scheduler's in-memory collection of all DAG objects parsed from configured DAG files. Parsing errors and import-time failures show up here and can silently remove DAGs.

Storage of a DAG's structure as JSON in the metadata database so the webserver and workers need not re-parse DAG files. It decouples UI rendering and task execution from file parsing.

Statements in a DAG file executed at import time rather than inside a task. Heavy or network-bound code here slows every parse cycle and should be avoided or deferred into callables.

A specific run of a task for one logical date and one DAG run, with its own state, try number, and logs. It is the unit the scheduler queues and the UI shows.

One execution of a DAG tied to a logical date, carrying its own state and configuration. Manual, scheduled, and triggered runs each create a separate record.

The bitshift syntax or set_upstream/set_downstream calls that declare order between tasks. It builds the edges of the DAG graph and determines what may run next.

A per-task setting controlling whether a task runs given the states of its upstream tasks, for example all_success, all_done, or one_failed. It shapes branching and cleanup behavior.

A visual and organizational grouping of tasks inside a DAG that can carry shared defaults and be nested. It keeps large graphs readable without creating a separate DAG.

Choosing which downstream path executes at runtime, typically with a branch operator returning task identifiers. Unchosen paths are marked skipped rather than failed.

Creating a variable number of task instances at runtime by expanding over an upstream list or XCom value. The graph is generated per run instead of being fixed at parse time.

A small key-value record that lets tasks exchange results through the metadata database. Values are meant to be small references, not large datasets, because they are serialized and stored.

Placeholder syntax in operator arguments evaluated at runtime against a context of dates, params, and XCom values. Templated fields are declared per operator and rendered before execution.

The timestamp identifying which data interval a DAG run covers, formerly called execution date. Templates and scheduling math treat it as the start of the interval, not the run time.

The time window a scheduled DAG run is responsible for, bounded by the logical date and the next schedule point. It clarifies which period of source data a run should process.

A DAG setting that makes the scheduler create runs for every missed interval since the start date. Disabling it limits execution to the most recent intervals only.

Deliberately running a DAG for past logical dates to reprocess historical intervals. It respects the same dependencies and can be limited to a date range.

The rule that decides when new DAG runs are created, given as a cron expression, timedelta, or timetable. It defines interval boundaries that determine logical dates.

A stored record of credentials and host details, referenced by a unique identifier so tasks need no hardcoded secrets. Hooks read connections to build clients.

A thin wrapper that turns a connection into a usable client for an external system, such as a database or cloud API. Operators call hooks internally to do their work.

A key-value setting stored centrally and read at runtime by tasks or templates. Useful for shared configuration, but frequent reads add database load.

An integration that lets Airflow fetch connections and variables from an external vault instead of the metadata database. It keeps sensitive values out of plain database storage.

Declared runtime inputs for a DAG or task, validated and accessible through templates and context. They make triggered runs configurable without editing code.

Automatic re-execution of a failed task up to a configured count, optionally with delay and backoff. Retries handle transient faults but require idempotent task logic.

A time threshold from the start of a DAG run by which a task should finish, used to raise alerts. It records misses but does not stop or retry the task.

A function attached to task or DAG lifecycle events such as failure or success, run to notify or clean up. Callbacks receive context and run outside the task's main callable.

The property that rerunning a task for the same interval yields the same result without duplicating side effects. It is essential for safe retries, backfills, and catchup runs.

A decorator-based style where plain Python functions become tasks and return values pass automatically as XComs. It reduces boilerplate for Python-centric pipelines.

A legacy way to embed a DAG factory inside a parent DAG as a single node. It is largely superseded by task groups because of scheduler and executor limitations.

The component that decides where and how queued task instances actually run, such as locally or across a cluster. It bounds parallelism and determines worker behavior.

The long-running service that parses DAG files, creates runs, and queues task instances whose dependencies are met. Its loop frequency and parsing cost dominate DAG performance.

A named slot limit that caps how many task instances of selected tasks may run at once. It protects shared resources from being overwhelmed by parallel tasks.

The relational store holding DAG structure, run and task states, XComs, connections, and variables. Contention or bloat here slows scheduling and UI responsiveness.

A check that imports a DAG file and asserts the graph loads and has expected tasks and edges. It catches import errors and structural mistakes before deployment.

Executing a task or DAG with no real side effects to verify wiring, templates, and arguments. It validates configuration and rendering without touching production systems.

A pluggable Python object that defines schedule intervals and logical dates beyond what cron can express. It gives full control over when runs are created and what interval they cover.