Airflow DAG Authoring study guide
Airflow DAG Authoring · Work through your exam, one topic at a time.
This page is an independent study reference for the Level 2 Apache Airflow 3 DAG Authoring certification offered by Astronomer, the company behind Astro. It is written for engineers who already write pipelines day to day: the vendor recommends roughly six or more months of hands-on Airflow experience, and the exam tests judgement about which Airflow feature to reach for, not just whether you can wire tasks together. The Airflow DAG Authoring study guide has 10 sections and about 3,122 words.
01DAG Fundamentals and Structure
An Airflow DAG is a Python object that declares what should run and when , not a record of past runs. Its identity is the dag id string plus the schedule; the same dag id in two files is a collision error, not two DAGs.
350 words, about 2 min02Operators and Task Construction
An operator is a template for a task; instantiating it creates a task in the active DAG. Know the families: action operators ( PythonOperator , BashOperator , SQLExecuteQueryOperator ), transfer operators, sensor operators ( ExternalTaskSensor , FileSensor ), and the decorator style ( @task , @task.bash ).
331 words, about 2 min03Task Dependencies and Control Flow
Dependencies are set with a b , a.set downstream(b) , or lists: [a, b] c means both a and b must succeed before c. chain(a, b, c) and cross downstream are helpers worth recognizing.
310 words, about 2 min04DAG Scheduling and Time Semantics
Airflow schedules by data interval . A run's logical date (formerly execution date ) is the start of the interval it processes.
304 words, about 2 min05Data Passing with XComs and Templating
XComs move small serializable values between tasks. A task pushes with task instance.xcom push(key, value) or by returning a value (stored under key return value ); downstream tasks pull with xcom pull(task ids=..., key=...) .
329 words, about 2 min06Connections, Hooks, and Variable Management
A Connection stores host, login, password, port, schema, and extra (JSON) under a conn id . Hooks are the programmatic interface: BaseHook.get connection('conn id') retrieves it, and specific hooks ( PostgresHook , S3Hook , HttpHook ) wrap it with client logic.
274 words, about 1 min07Dynamic DAGs and Task Mapping
Dynamic DAG generation produces many DAGs from configuration - a loop over a YAML file calling a factory function, each iteration returning a DAG with a unique dag id . Dynamic task mapping produces a variable number of task instances at runtime from .expand() or .partial() .
307 words, about 2 min08Error Handling, Retries, and SLAs
Tasks end in success , failed , upstream failed , skipped , or up for retry . retries and retry delay control automatic re-attempts; retry exponential backoff and max retry delay shape the schedule.
291 words, about 1 min09DAG Testing, Debugging, and Performance
Airflow offers airflow dags test <dag id <logical date (single-run, no scheduler), airflow tasks test <dag id <task id <date (runs one task in isolation), and airflow dags list-import-errors . For CI, use DagBag to load DAG files and assert no import errors, plus dag.test() for in-process runs.
316 words, about 2 min10Best Practices and DAG Hygiene
Production DAGs should be idempotent : re-running an interval produces the same result, which means writing to partitioned destinations keyed by {{ ds }} and using MERGE/overwrite rather than blind appends. Keep DAG files thin: put logic in importable modules under plugins/ or a package, and let the DAG file declare structure only.
310 words, about 2 min