SLA Misses Without Alerts
A critical DAG has sla=timedelta(hours=2) on its final load task. The DAG runs every 30 minutes. The team reports that the SLA has been missed several times but no alerts were received. The Airflow deployment uses the default sla_miss_callback=None and email is not configured. The load task has retries=3 and retry_delay=timedelta(minutes=20). The scheduler has scheduler_health_check_threshold at defaults. The team also wants retries to not count against the SLA.
1. Explain why SLA misses produced no alerts and how retries interact with SLA timing.
Read the worked answer
SLA misses are recorded in the metadata database and surfaced in the UI, but notifications require a callback or email configuration. With sla_miss_callback=None and no email backend, misses are silent. The team must set sla_miss_callback to a function that sends to their alerting system, or configure email on the DAG.
SLA timing is measured from the start of the DAG run's logical interval, not from when the task starts, and retries extend the wall-clock time the task occupies. With three retries at 20 minutes each, the task can consume over an hour before succeeding, which can push past the 2-hour SLA. Airflow does not natively exclude retry time from SLA calculation; the SLA is a deadline check, not a duration-of-successful-attempt check. To avoid counting retries, the team must either reduce retry delay, move the SLA to a task that is not retried, or compute a custom deadline.
Also note that SLA checks are evaluated by the scheduler and can be delayed under load, so the alert may lag.
2. Propose a revised alerting and SLA strategy that gives the team reliable notification and accurate semantics.
Read the worked answer
Implement an sla_miss_callback that posts to the team's incident tool with DAG ID, task ID, logical date, and duration. Configure email as a fallback if the deployment supports it. Test the callback by forcing an SLA miss in a staging DAG.
For semantics, decide whether the SLA is about completion deadline or successful-attempt duration. If it is a deadline, keep sla on the load task but account for retry budget in the value, for example set SLA to 2 hours and ensure retries plus delay fit within it. If it is about successful duration, use a separate monitoring task that measures elapsed time and alerts, rather than relying on Airflow SLA.
Add a sensor or check that the DAG run started on time, since a late start also causes misses. Document the SLA definition and review it after each incident.