Airflow DAG Authoring study plan
Airflow DAG Authoring · Give each week a clear focus.
8 weeks · About 6 hours per week
DAG fundamentals and structure & Operators and task construction
Write a minimal DAG file from scratch and explain each top-level component (DAG object, default_args, operator instantiation) without notes
Convert a script-style pipeline into a DAG with four BashOperator and PythonOperator tasks and verify it parses with a DAG bag load
Instantiate the DAG using a context manager and again using explicit dag= arguments, then state two differences in behavior
Set start_date, schedule, catchup, and tags on a DAG and articulate what each controls
Your checkpoint
A working two-DAG repository (one context-manager style, one explicit-argument style) that parses cleanly and passes a DAG-bag import check.
Operators and task construction & Task dependencies and control flow
Build tasks with PythonOperator, BashOperator, and at least one provider operator, passing arguments correctly to each
Express a fan-out/fan-in dependency graph using both bitshift operators and set_upstream/set_downstream, then compare the two notations
Implement branching with BranchPythonOperator and confirm skipped-branch behavior including trigger rules
Add a TaskGroup to organize related tasks and verify the rendered graph structure
Your checkpoint
A single DAG with a branch, a fan-out/fan-in section, and a TaskGroup, plus a written note explaining which tasks skip and why under each trigger rule used.
DAG scheduling and time semantics & Data passing with XComs and templating
Distinguish logical date, run date, and data interval, and predict the execution_date a daily DAG receives for a given schedule
Configure a DAG with catchup disabled and explain how backfill differs from normal scheduled runs
Push and pull values with XComs using both the return-value convention and explicit xcom_push/xcom_pull
Template a BashOperator command and a PythonOperator argument with ds, data_interval_start, and params, then verify rendering in a task log
Your checkpoint
A daily DAG with templated commands and one XCom handoff, plus a table mapping three logical dates to their data intervals and rendered template values.
Dynamic DAGs and task mapping & Task dependencies and control flow
Generate multiple DAGs or tasks programmatically from a config list and explain why top-level loop variables must be bound safely
Convert a repeated-task pattern into a dynamically mapped task using expand and partial
Control mapped-task concurrency with max_active_tis_per_dag or map-length limits and describe the effect
Combine mapped tasks with a downstream aggregation task and confirm dependency semantics
Your checkpoint
A parameterized DAG factory that produces at least three DAGs from one config, one of which uses dynamic task mapping, with a short README explaining the expansion logic.
Connections, hooks, and variable management & Data passing with XComs and templating
Create and reference an Airflow connection by conn_id, then retrieve it in a task through a hook rather than hardcoding credentials
Store and read Airflow Variables at DAG-parse time versus task runtime, and explain the performance implication of each
Pass a small config object through params and templating instead of XCom where appropriate, and justify the choice
Identify three cases where XCom is the wrong tool and propose an alternative for each
Your checkpoint
A DAG that reads a connection and a Variable, uses a hook to perform a trivial operation, and includes a written decision log of XCom versus params versus external storage.
Error handling, retries, and SLAs & DAG scheduling and time semantics
Configure retries, retry_delay, and exponential backoff on tasks and predict the resulting attempt timeline
Implement on_failure_callback and on_retry_callback and log structured failure context from each
Set an SLA and explain which timestamp it is measured against and what happens when it is missed
Choose appropriate trigger rules for an error-handling task that must run only after upstream failure
Your checkpoint
A DAG with per-task retry policies, failure and retry callbacks, an SLA, and an all_done cleanup task, accompanied by a timeline diagram of expected attempt times.
DAG testing, debugging, and performance & DAG fundamentals and structure
Write DagBag-based tests that assert task count, dependency edges, and absence of import errors
Test a PythonOperator callable in isolation with mocked hooks or connections
Use airflow tasks test to run a single task locally and interpret the output
Diagnose a slow DAG parse caused by top-level database or network calls and refactor it
Your checkpoint
A pytest suite of at least six tests covering DAG structure and one mocked task callable, all passing locally, plus a before/after parse-time measurement.
Best practices and DAG hygiene & Error handling, retries, and SLAs
Audit a DAG against a hygiene checklist covering naming, idempotency, no top-level side effects, and secrets handling
Refactor a DAG to remove duplicated boilerplate using default_args or a shared factory
Verify idempotency by reasoning through a rerun of the same data interval and listing any non-idempotent steps
Produce a final review of all practice DAGs and correct at least three defects found
Your checkpoint
A refactored capstone DAG plus a completed hygiene checklist documenting each fix and the reasoning behind it.
Milestones
Week 2: Can build and parse a multi-task DAG with branching and TaskGroups from a blank file in under 20 minutes without references.
Week 4: Can explain logical date versus data interval correctly for three schedule types and demonstrate a working dynamically mapped task.
Week 6: Can predict a retry timeline and configure callbacks and SLAs correctly on a first attempt.
Week 8: Passes the full pytest suite and completes the hygiene checklist with no unresolved defects in the capstone DAG.