Airflow DAG Authoring case studies
Airflow DAG Authoring · Put your knowledge to work in context.
This page is an independent study reference for the Level 2 Apache Airflow 3 DAG Authoring certification offered by Astronomer, the company behind Astro. It is written for engineers who already write pipelines day to day: the vendor recommends roughly six or more months of hands-on Airflow experience, and the exam tests judgement about which Airflow feature to reach for, not just whether you can wire tasks together. The syllabus is anchored in the Airflow 3 era - assets and asset-aware scheduling, dynamic task mapping, dynamic DAGs, DAG versioning and DAG bundles, DAG scheduling, the TaskFlow API, TaskGroups, templating, XComs, and branching. Expect scenario-style items where more than one approach would technically run, but only one is reliable, maintainable, or correct under the documented semantics. The Airflow DAG Authoring case studies are 5 cases and about 2,164 words.
1The Backfill That Re-Ran Everything
A team runs an hourly DAG with start date=datetime(2024,1,1) , schedule='@hourly' , catchup=True , and no max active runs . The DAG has run for 14 months.
450 words, about 2 min2XCom Payload Bloat
A pipeline extracts a 2 GB Parquet file from object storage, then passes the file path and a 40 MB summary dictionary to a downstream task via xcom push . After two weeks, the metadata database has grown by 30 GB and the scheduler UI is slow.
438 words, about 2 min3Dynamic Task Mapping With Uneven Batches
A DAG uses @task with .expand() to process a list of customer IDs fetched at runtime. The list varies from 50 to 5,000 IDs.
428 words, about 2 min4SLA Misses Without Alerts
A critical DAG has sla=timedelta(hours=2) on its final load task. The DAG runs every 30 minutes.
419 words, about 2 min5Testing a DAG Without a Cluster
A team wants CI to catch broken DAGs before deploy. Their DAGs import a custom internal library, call Variable.get() at module import time, and use PostgresHook inside task functions.
429 words, about 2 min