DAG Fundamentals and Structure
What a DAG really is
An Airflow DAG is a Python object that declares what should run and when, not a record of past runs. Its identity is the dag_id string plus the schedule; the same dag_id in two files is a collision error, not two DAGs. Understand the core attributes: schedule (a cron string, timedelta, timetable, or None), start_date, catchup, max_active_runs, default_args, tags, and doc_md.
Tasks are nodes; edges come from >> and <<. A DAG with no edges still runs all tasks in parallel. Every task needs a unique task_id within its DAG.
Traps candidates hit
- start_date semantics: the first run covers the interval starting at
start_date, so withschedule='@daily'andstart_date=2024-01-01, the run for logical date 2024-01-01 fires at the end of that day. default_argsis a dict of fallbacks, not a schema; values are copied into tasks, and mutable values shared across tasks cause surprising coupling.- DAG context managers are the modern idiom; the older
DAG(...)assigned to a variable still works but makes accidental cross-DAG task attachment easier. - Top-level code runs on every parse, so heavy imports or API calls at module scope slow the scheduler.
Rule of thumb
Treat the DAG file as a declaration: only definitions, no side effects, no I/O at import time. If a task must be created conditionally, do it with a factory function that returns the DAG object, not with if statements that mutate a global DAG.
Worked example
with DAG('etl', start_date=datetime(2024,1,1), schedule='@daily', catchup=False, default_args={'retries':2}) as dag: then define tasks inside. Setting catchup=False prevents backfilling every missed interval since January.