Airflow DAG Authoring study materials
Airflow DAG Authoring study resources

Airflow DAG Authoring study plan

Airflow DAG Authoring · Give each week a clear focus.

8 weeks · About 6 hours per week

Week 1

DAG fundamentals and structure & Operators and task construction

  • Write a minimal DAG file from scratch and explain each top-level component (DAG object, default_args, operator instantiation) without notes

  • Convert a script-style pipeline into a DAG with four BashOperator and PythonOperator tasks and verify it parses with a DAG bag load

  • Instantiate the DAG using a context manager and again using explicit dag= arguments, then state two differences in behavior

  • Set start_date, schedule, catchup, and tags on a DAG and articulate what each controls

Your checkpoint

A working two-DAG repository (one context-manager style, one explicit-argument style) that parses cleanly and passes a DAG-bag import check.

Week 2

Operators and task construction & Task dependencies and control flow

  • Build tasks with PythonOperator, BashOperator, and at least one provider operator, passing arguments correctly to each

  • Express a fan-out/fan-in dependency graph using both bitshift operators and set_upstream/set_downstream, then compare the two notations

  • Implement branching with BranchPythonOperator and confirm skipped-branch behavior including trigger rules

  • Add a TaskGroup to organize related tasks and verify the rendered graph structure

Your checkpoint

A single DAG with a branch, a fan-out/fan-in section, and a TaskGroup, plus a written note explaining which tasks skip and why under each trigger rule used.

Week 3

DAG scheduling and time semantics & Data passing with XComs and templating

  • Distinguish logical date, run date, and data interval, and predict the execution_date a daily DAG receives for a given schedule

  • Configure a DAG with catchup disabled and explain how backfill differs from normal scheduled runs

  • Push and pull values with XComs using both the return-value convention and explicit xcom_push/xcom_pull

  • Template a BashOperator command and a PythonOperator argument with ds, data_interval_start, and params, then verify rendering in a task log

Your checkpoint

A daily DAG with templated commands and one XCom handoff, plus a table mapping three logical dates to their data intervals and rendered template values.

Week 4

Dynamic DAGs and task mapping & Task dependencies and control flow

  • Generate multiple DAGs or tasks programmatically from a config list and explain why top-level loop variables must be bound safely

  • Convert a repeated-task pattern into a dynamically mapped task using expand and partial

  • Control mapped-task concurrency with max_active_tis_per_dag or map-length limits and describe the effect

  • Combine mapped tasks with a downstream aggregation task and confirm dependency semantics

Your checkpoint

A parameterized DAG factory that produces at least three DAGs from one config, one of which uses dynamic task mapping, with a short README explaining the expansion logic.

Week 5

Connections, hooks, and variable management & Data passing with XComs and templating

  • Create and reference an Airflow connection by conn_id, then retrieve it in a task through a hook rather than hardcoding credentials

  • Store and read Airflow Variables at DAG-parse time versus task runtime, and explain the performance implication of each

  • Pass a small config object through params and templating instead of XCom where appropriate, and justify the choice

  • Identify three cases where XCom is the wrong tool and propose an alternative for each

Your checkpoint

A DAG that reads a connection and a Variable, uses a hook to perform a trivial operation, and includes a written decision log of XCom versus params versus external storage.

Week 6

Error handling, retries, and SLAs & DAG scheduling and time semantics

  • Configure retries, retry_delay, and exponential backoff on tasks and predict the resulting attempt timeline

  • Implement on_failure_callback and on_retry_callback and log structured failure context from each

  • Set an SLA and explain which timestamp it is measured against and what happens when it is missed

  • Choose appropriate trigger rules for an error-handling task that must run only after upstream failure

Your checkpoint

A DAG with per-task retry policies, failure and retry callbacks, an SLA, and an all_done cleanup task, accompanied by a timeline diagram of expected attempt times.

Week 7

DAG testing, debugging, and performance & DAG fundamentals and structure

  • Write DagBag-based tests that assert task count, dependency edges, and absence of import errors

  • Test a PythonOperator callable in isolation with mocked hooks or connections

  • Use airflow tasks test to run a single task locally and interpret the output

  • Diagnose a slow DAG parse caused by top-level database or network calls and refactor it

Your checkpoint

A pytest suite of at least six tests covering DAG structure and one mocked task callable, all passing locally, plus a before/after parse-time measurement.

Week 8

Best practices and DAG hygiene & Error handling, retries, and SLAs

  • Audit a DAG against a hygiene checklist covering naming, idempotency, no top-level side effects, and secrets handling

  • Refactor a DAG to remove duplicated boilerplate using default_args or a shared factory

  • Verify idempotency by reasoning through a rerun of the same data interval and listing any non-idempotent steps

  • Produce a final review of all practice DAGs and correct at least three defects found

Your checkpoint

A refactored capstone DAG plus a completed hygiene checklist documenting each fix and the reasoning behind it.

Milestones

Week 2: Can build and parse a multi-task DAG with branching and TaskGroups from a blank file in under 20 minutes without references.

Week 4: Can explain logical date versus data interval correctly for three schedule types and demonstrate a working dynamically mapped task.

Week 6: Can predict a retry timeline and configure callbacks and SLAs correctly on a first attempt.

Week 8: Passes the full pytest suite and completes the hygiene checklist with no unresolved defects in the capstone DAG.