The Backfill That Re-Ran Everything
A team runs an hourly DAG with start_date=datetime(2024,1,1), schedule='@hourly', catchup=True, and no max_active_runs. The DAG has run for 14 months. A developer adds a new task export_to_warehouse midway through the file and deploys on a Tuesday afternoon. Within minutes the scheduler queues thousands of runs. The warehouse team reports duplicate exports for old dates. The DAG's end_date is unset and depends_on_past=False. The developer expected only new runs to include the export. The team has a 2-hour window to stop the bleeding before the warehouse load window.
1. Explain why thousands of runs were queued and why duplicate exports appeared for historical dates.
Read the worked answer
The scheduler creates a DagRun for every interval between start_date and now when catchup=True and no end_date exists. With hourly scheduling over 14 months, that is roughly 10,000 intervals. Adding a task does not change the schedule, but the newly deployed DAG file is parsed and the scheduler reconciles: existing runs already completed, but the backfill behavior is governed by catchup, not by whether the task is new.
Duplicate exports appear because the export task is not idempotent with respect to the logical date. When runs are created or re-created, the task executes for each historical interval. If the export writes rows keyed only by execution time of the worker rather than {{ ds }} or {{ data_interval_start }}, the same interval can be written twice. depends_on_past=False removes any serialization that might have slowed the flood.
The fix is to pause the DAG immediately, set catchup=False (or set end_date), and make the export idempotent using the logical date in the key. Then clear only the intended runs.
2. Describe the immediate mitigation steps and the durable configuration changes you would make, in order.
Read the worked answer
Immediate mitigation: pause the DAG in the UI or via airflow dags pause, then stop the scheduler if necessary to halt queue growth. Identify affected intervals by querying the metadata database for DagRuns created after the deploy timestamp. Notify the warehouse team with the interval list so they can deduplicate.
Durable changes: set catchup=False for the DAG if historical backfill is not desired, or set a bounded end_date. Set max_active_runs to a small number such as 2-3 to cap concurrency. Make the export idempotent by writing with a partition key derived from {{ data_interval_start }} and using MERGE or overwrite semantics. Add a max_active_tasks limit.
For the historical repair, run a controlled backfill with airflow dags backfill --start-date ... --end-date ... after deduplication, rather than relying on scheduler catchup. Document the change in the DAG docstring.