Airflow DAG Authoring flashcards

38 cards. Read the term and the definition here. The deck at the top of this page is optional.

DAG dag_id

A unique string identifying a DAG within an Airflow deployment. The scheduler stores runs and task instances under this key, so duplicates across files cause conflicts.

start_date requirement

The DAG's start_date must be a timezone-aware datetime, typically pendulum.datetime(..., tz='UTC'). Naive datetimes are rejected or coerced, shifting schedule boundaries unexpectedly.

default_args dict

A dictionary of operator defaults merged into every task in the DAG. Explicit operator keyword arguments override it, and values like retries or owner apply to all tasks.

catchup flag

When True, the scheduler creates runs for every interval between start_date and now. When False, it skips missed intervals and schedules only the most recent one.

BashOperator

An operator that runs a shell command on a worker. Its bash_command field is templated, so Jinja expressions such as {{ ds }} render before execution.

template_fields

A tuple on an operator class listing attribute names that Jinja renders at runtime. Only fields in this list receive templated context values.

Sensor reschedule mode

A sensor setting where the worker frees its slot between pokes instead of holding it. It reduces slot consumption for long waits at the cost of more scheduler work.

PythonOperator callable

The python_callable argument names the function to run when the task executes. Context values are passed only if the function signature accepts them.

Bitshift dependency syntax

The >> and << operators set task order: a >> b runs a before b. Lists fan in or out, so [a, b] >> c waits for both upstream tasks.

TaskGroup

A UI grouping construct that namespaces member task_ids as group_id.task_id. Dependencies can still be set across group boundaries using the task objects.

BranchPythonOperator

An operator whose callable returns one or more task_ids. The scheduler follows those branches and marks all other downstream branches as skipped.

trigger_rule none_failed_min_one_success

A join rule that runs a task when at least one upstream succeeded and none failed. It is common after branching, where unselected branches are skipped.

Logical date

The start timestamp of the interval a DAG run covers, formerly called execution_date. It is not the wall-clock time the run actually executed.

Data interval

The window bounded by data_interval_start and data_interval_end that a run processes. For cron schedules it spans from one tick to the next.

depends_on_past

A task setting that blocks the current instance until the same task succeeded in the previous DAG run. It enforces sequential processing across intervals.

max_active_runs

A DAG-level limit on concurrently running DAG runs. It bounds parallelism and is often paired with catchup to avoid overwhelming downstream systems.

XCom push and pull

XComs store small JSON-serializable values in the metadata database. Tasks push with xcom_push or a return value and pull with xcom_pull using task_ids and key.

Jinja ds variable

A templating context variable holding the logical date as YYYY-MM-DD. The ds_nodash variant omits dashes, and ts gives a full ISO-8601 timestamp.

XComArg task output

The object returned by a taskflow task or task.output, representing a future XCom value. It can be passed as an argument or into expand for mapping.

render_template_as_native_obj

A DAG flag that keeps rendered template results as native Python types such as dict or list instead of strings. It helps when downstream code expects structured data.

Connection conn_id

A named record storing host, schema, login, password, port, and extra. Hooks and operators reference it by conn_id to obtain credentials without hardcoding them.

Hook class

A BaseHook subclass that reads a Connection and exposes a client for an external system. Operators typically build a hook internally from the conn_id argument.

Variable.get caching caution

Calling Variable.get at module level queries the database on every DAG parse. Templated var.value references or params avoid this parse-time cost.

Fernet key

A configuration secret used to encrypt Connection passwords and Variable values at rest in the metadata database. Without it, secrets are stored in plaintext.

Dynamic task mapping expand

The expand method creates one task instance per element of a list or XComArg at runtime. The instance count is determined when the upstream value is available.

partial freezes arguments

The partial method fixes arguments on a mapped task so only the expanded argument varies per instance. It is combined with expand on the same task.

max_mapped_tasks

A DAG-level cap on the number of task instances a single mapped task may create. Exceeding the cap fails the mapped task at runtime.

Module-level work caution

Code at the top level of a DAG file runs on every scheduler parse. API calls or database queries there slow parsing and should be moved into tasks or cached.

retries and retry_delay

Operator arguments setting how many times a failed task retries and how long to wait between attempts. retry_exponential_backoff increases the delay each attempt.

on_failure_callback

A callable invoked with the task context when a task fails. It runs in the scheduler or worker process, so it should be fast and free of heavy work.

SLA miss behavior

A task sla=timedelta setting triggers an sla_miss_callback and logs an event when exceeded. It does not stop or fail the running task.

AirflowSkipException

Raising this exception inside a callable marks the task as skipped rather than failed. Downstream tasks then follow their trigger rules for skipped upstreams.

DagBag validation

A class that loads DAG files from a folder and exposes a dags dictionary. Tests assert expected dag_ids are present to catch import errors before deployment.

airflow tasks test

A CLI command that runs a single task locally for a given logical date without the scheduler. It is used to debug task logic and templating.

Grid view

A UI page showing DAG runs over time and task states per run. It helps spot failures, retries, and scheduling gaps across intervals.

Idempotent DAG design

A design principle where rerunning a task for the same interval produces the same result. It relies on overwrite or upsert logic rather than appending duplicates.

execution_timeout

An operator argument that kills a running task after the given duration. It prevents hung processes from occupying worker slots indefinitely.

Pool and pool_slots

A pool is a named concurrency limit shared across tasks. pool_slots sets how many units a task consumes, protecting external systems from overload.