Airflow DAG Authoring: Best Practices and DAG Hygiene

Idempotence, modularity, and clarity

Production DAGs should be idempotent: re-running an interval produces the same result, which means writing to partitioned destinations keyed by {{ ds }} and using MERGE/overwrite rather than blind appends. Keep DAG files thin: put logic in importable modules under plugins/ or a package, and let the DAG file declare structure only.

Hygiene checklist: meaningful dag_id and tags, doc_md explaining ownership, owner in default_args, consistent retries, catchup=False unless backfill is intended, no secrets in code, small XComs, and one responsibility per task. Use TaskGroup for visual grouping without changing dependencies. Version-control DAGs, review them, and delete or pause retired ones.

Traps candidates hit

• Non-idempotent tasks that append rows duplicate data on retry or backfill.
• Hardcoded dates instead of templated {{ ds }} break backfills.
• Giant "do everything" Python tasks hide failures and make retries expensive.
• catchup=True by default thinking causes accidental floods; be explicit.
• Duplicate dag_ids across files silently shadow each other and cause confusing behavior.
• Missing owner makes alert routing impossible.

Rule of thumb

Ask two questions of every task: "If this runs twice for the same interval, is the outcome the same?" and "Could someone else understand and fix this from the DAG file alone?" If either answer is no, refactor.

Worked example

A load task writes INSERT OVERWRITE TABLE t PARTITION(ds='{{ ds }}') SELECT ..., so retries replace the partition rather than duplicating it. The DAG file contains only operator declarations and a TaskGroup('load'); transformation SQL lives in a versioned file.