Airflow Fundamentals flashcards
Airflow Fundamentals · Build recall with a focused review session.
33 cards. Read the term and the definition here. The deck at the top of this page is optional.
- Grid view
- A UI screen showing a matrix of DAG runs by tasks, where each colored cell represents one task instance state for that run.
- Task instance states
- Values describing a single task attempt, including scheduled, queued, running, success, failed, up_for_retry, skipped, and up_for_reschedule.
- dags list-import-errors
- An Airflow CLI command that prints DAG files the scheduler failed to parse, exposing Python syntax or import errors hidden from the UI.
- tasks test command
- An Airflow CLI command that runs one task locally for a given logical date without writing state to the metadata database.
- Backfill
- Creating DAG runs for historical intervals, typically via airflow dags backfill with start and end dates, to reprocess past data.
- When Airflow is a poor fit
- Streaming with sub-second latency, always-on services, or per-record loops; Airflow is a batch orchestrator that triggers external systems.
- catchup parameter
- When True, the scheduler creates runs for every interval since start_date; when False, it skips directly to the most recent interval.
- Logical date
- The timestamp labeling the start of a DAG run's data interval, formerly called execution_date; it is not the wall-clock execution time.
- ExternalTaskSensor
- A sensor that waits for a task in another DAG to reach a target state, requiring external_dag_id, external_task_id, and aligned execution dates.
- TriggerDagRunOperator
- An operator that starts another DAG from within a task, optionally passing a conf dictionary to the triggered run.
- Dataset-triggered DAG run
- A run created when a producer task declares an outlet dataset and a consumer DAG schedules on that dataset instead of a time interval.
- DAG run identity
- A run is uniquely identified by dag_id plus logical date, so the same interval cannot execute twice simultaneously.
- Scheduler role
- The component that parses DAG files, creates DAG runs on schedule, and queues task instances whose dependencies are satisfied.
- CeleryExecutor
- A distributed executor that sends tasks to worker processes through a broker such as Redis or RabbitMQ, enabling parallel execution across hosts.
- Metadata database
- The central store, PostgreSQL or MySQL in production, holding DAG definitions, run and task states, variables, connections, and XCom values.
- PythonOperator
- An operator that runs a Python callable specified by python_callable, passing positional and keyword arguments via op_args and op_kwargs.
- EmptyOperator
- An operator that performs no action, used to join branches or fan out dependencies in a DAG graph.
- BranchPythonOperator
- An operator whose callable returns a task_id; that branch runs while unselected downstream branches are marked skipped.
- XCom default key
- When a task returns a value, it is pushed under the key return_value, retrievable with xcom_pull using that key.
- XCom size guidance
- XCom suits small JSON-serializable metadata like IDs and paths; large payloads should go to object storage with only the path passed.
- Connection object
- A stored record of host, login, password, schema, port, and JSON extra, referenced by conn_id to build hooks and clients.
- Variable.get at parse time
- Calling Variable.get at DAG top level queries the database on every scheduler parse; template {{ var.value.key }} avoids that cost.
- Gantt view
- A UI timeline plotting task start and end times for a run, revealing queueing delays and long-running tasks.
- UI Code view
- A read-only display of the DAG file source; changes appear only after the scheduler reparses the file.
- airflow tasks run
- An Airflow CLI command that executes a task for a given date and does persist state, unlike the tasks test command.
- airflow db migrate
- An Airflow CLI command that creates or upgrades the metadata database schema to match the installed Airflow version.
- Idempotent tasks
- Tasks that produce the same result when rerun for the same interval, making backfills and clears safe to repeat.
- default_args
- A dictionary of shared task settings such as owner, retries, and retry_delay applied to all tasks in a DAG unless overridden.
- Task dependency syntax
- Dependencies are declared with a >> b or a << b bit-shift operators, or with set_downstream and set_upstream method calls.
- TaskGroup
- A container that groups tasks visually and for dependency wiring inside a DAG without changing execution semantics.
- Trigger rules
- Rules such as all_success, all_failed, one_success, and none_failed that decide when a task runs based on upstream states.
- Triggerer component
- An Airflow process that runs deferrable operators and sensors asynchronously, freeing worker slots while tasks wait.
- TaskFlow API
- A decorator-based style where @task wraps Python functions as tasks and return values pass between tasks implicitly through XCom.