DAG Testing, Debugging, and Performance
Test the DAG, not just the task
Airflow offers airflow dags test <dag_id> <logical_date> (single-run, no scheduler), airflow tasks test <dag_id> <task_id> <date> (runs one task in isolation), and airflow dags list-import-errors. For CI, use DagBag to load DAG files and assert no import errors, plus dag.test() for in-process runs. Unit-test callables directly with mocked hooks.
Debugging tools: task logs, the grid and graph views, airflow tasks state, and airflow tasks clear to re-run. The scheduler parses DAG files repeatedly, so parse time is a first-class performance metric; airflow dags report and the dag_processor logs reveal slow files.
Traps candidates hit
airflow tasks testignores dependencies and does not write to the metadata DB the same way, so downstream XComs may be missing.- Top-level DB calls, API calls, or heavy imports slow every parse; move them inside callables.
- Too many small tasks increases scheduler overhead; too few large tasks reduces observability.
- Wide fan-out with default pools starves other DAGs; use pools and
max_active_tasks. - Testing with
catchup=Truebackfills and surprises CI. dag.test()runs in-process, so it needs a configured environment but no scheduler.
Rule of thumb
A DAG that imports in under a second and has no module-level I/O is already fast. Test import health in CI before testing behavior.
Worked example
CI step: build a DagBag(dag_folder='dags/'), assert len(dagbag.import_errors) == 0, then for each DAG assert dag.timetable and task ids match expectations. This catches syntax errors, duplicate ids, and accidental schedule changes cheaply.