Writing, Scheduling & Debugging DAGs
Structure
A DAG is a Python file that defines a DAG object (or uses @dag) with a dag_id, start_date, schedule, and catchup setting. Tasks are operator instances; dependencies are set with >> or set_upstream/set_downstream. Keep the top-level file cheap: the scheduler imports it on a loop, so avoid network calls, heavy queries, or unbounded loops at module level.
Scheduling semantics
start_date plus schedule defines the run schedule. For a daily schedule starting 2024-01-01, the run with logical_date 2024-01-01 covers the interval ending at that time and is triggered at the end of the interval - a classic trap. catchup=True (the default in many versions) creates runs for every missed interval since start_date; setting catchup=False limits history to recent runs. schedule=None means manual triggering only. Use max_active_runs and max_active_tasks to bound concurrency.
Debugging
Start with the task log in the UI. Test logic locally with airflow tasks test. Use depends_on_past and retries deliberately. Common failures: import errors (DAG missing entirely), timezone-naive datetimes, template fields not rendering because the operator lacks template_fields, and tasks stuck in queued because of pool limits or no free worker slots.
Rule of thumb
If a DAG vanishes, suspect a parse error first; if a task never starts, suspect scheduling, dependencies, or pool capacity - not the code inside the task.