The Sensor That Ate the Worker Pool

The twelve sensors in mode='poke' each occupy a worker slot for their entire lifetime, because poking is a loop inside a running task. With total worker_concurrency=16 and twelve sensors holding slots from 02:00, only four slots remain. Any other task that becomes runnable - including the entire 02:15 DAG - waits in queued until a slot frees, which happens only when the partitions appear around 02:30–03:00. That is precisely the observed pattern: the unrelated DAG is starved by sensors it has no relationship with.

Changes, in order of impact:

1. mode='reschedule' on every sensor. The task exits after each check and re-queues itself, releasing the slot between pokes. Twelve sleeping sensors then consume roughly zero steady-state slots instead of twelve. This alone fixes the starvation.
2. Set a timeout. Without one, a partition that never arrives holds a slot forever in poke mode and hangs the DAG run indefinitely. A timeout of, say, 2 hours converts an infinite hang into a visible failure with a retry opportunity.
3. Tune poke_interval. 60 seconds is fine for reschedule mode; for poke mode, longer intervals reduce database load but lengthen the hold. The interval should reflect how quickly the pipeline needs to react, not be a default.
4. Consider max_active_tis_per_dag / pool limits so a single DAG cannot monopolize the pool; a dedicated pool for sensors with a small slot count is a common guardrail.

Note that raising worker_concurrency treats the symptom and scales the cost with the number of sensors; reschedule mode removes the consumption.

In reschedule mode the sensor is not continuously watching. After each check it exits, and the scheduler must re-queue it at the next poke_interval. So detection latency is bounded by poke_interval plus scheduler loop time and queue latency. With poke_interval=60 you would expect detection within roughly a minute or two, not eleven.

Likely causes to check, in order:

• poke_interval is not actually 60 on this DAG (a default or an override elsewhere), or exponential_backoff is enabled, which lengthens intervals after repeated failures and can easily produce multi-minute gaps.
• Scheduler latency: a busy scheduler with many DAG files, a slow metadata database, or a long min_file_process_interval delays re-queuing. Check scheduler health and DAG parse times in the UI.
• Queue/worker latency: once queued, the task waits for a free worker slot; if the pool is saturated by other work at 02:20, the delay is scheduling, not sensing.
• The partition check itself is slow (a metastore query taking tens of seconds), which adds to each cycle.

The trade-off to articulate: reschedule mode trades detection latency and metadata-database churn (one task instance state transition per poke) for worker-slot efficiency. Poke mode trades slot occupancy for lower latency. For a partition expected within a 30-minute window, reschedule with a 60-second interval and backoff disabled is the usual right answer; if sub-minute latency were genuinely required, a push-based trigger (an event or an external task sensor on the producing DAG) is better than tightening the poll.

It depends on whether the partitions are produced by a known Airflow DAG with a matching schedule.

Good fit if: a single upstream DAG creates the partitions, it has the same logical-date cadence, and the team controls it. Then ExternalTaskSensor expresses the real dependency ('wait for upstream's load task for the same logical date') and is cheaper than polling an external system. Key details to get right: external_dag_id, external_task_id, and crucially execution_date_fn or allowed_states - by default it looks for the same logical date, which is wrong if the upstream runs on a different cadence or timezone, and is a classic source of sensors that wait forever. It should also run in reschedule mode with a timeout.

Poor fit if: partitions are created by something outside Airflow (a vendor, a Kafka consumer, another orchestrator), or by many upstream DAGs, or on an irregular cadence. Then there is no single task instance to wait on, and a metastore sensor is the honest representation. Forcing an ExternalTaskSensor would require a fake upstream DAG purely to satisfy the sensor, which couples unrelated systems.

Also note the failure mode difference: a metastore sensor fails when the data is absent; an ExternalTaskSensor fails when the upstream DAG run is absent or in a failed state, which may be a different operational problem with a different on-call owner. Whichever is chosen, set timeout, use reschedule mode, and make sure the downstream DAG's start_date and schedule align with upstream so the sensor is not waiting for a run that will never exist.