SLOs in a Spreadsheet, 300 Alerts a Week, No Pager
Forty product teams at Meridian use Datadog. SLOs live in a spreadsheet and are reviewed monthly; incidents are declared ad hoc in Slack threads with no named owner; there is no on-call rotation in Datadog yet because PagerDuty is being retired next quarter. Each team runs 8-15 monitors with static thresholds, producing roughly 300 notifications a week, about 80% of which nobody acts on. Checkout handles 3,000 requests per minute against a 99.9% availability target; the batch settlement job runs once nightly. Leadership wants error budgets that engineers actually use and fewer, better alerts.
1. Choose an SLO type for checkout and for the settlement job, and justify both.
Read the worked answer
Checkout: metric-based. At 3,000 requests/minute there are roughly 130 million events in a 30-day window, far more than enough for a 99.9% target to be statistically meaningful, and a good/total ratio computed from trace or log counts reacts to real user pain within minutes. Define the denominator narrowly - only the endpoints that constitute 'checkout available' - or a shift in traffic mix will silently move the number. Alert on error-budget burn rate: fast burn (e.g. 14x over 1 hour) pages, slow burn (e.g. 2x over 6 hours) opens a ticket.
Settlement: monitor-based. One run per night is one data point. A metric-based 99% target over 30 days would be decided by a single failure and would swing between 0% and 100% - statistically meaningless and guaranteed to be ignored. Wrap the existing job monitor (success/failure, or duration) in a monitor-based SLO, and add a separate freshness check for 'did it start at all', because a job that never runs is not a failed job.
Both need an owner. Register the service in the Service Catalog so each SLO has a team attached and a dashboard a director can open. An SLO with no owner is the spreadsheet with extra steps, and error-budget policy only works when there is someone to enforce it.
2. Cut the 300 weekly notifications to something responders act on. What changes do you make and in what order?
Read the worked answer
Classify before you tune. For two weeks, tag each notification: fired / actioned / which incident it fed. In most estates, 80% of noise comes from a handful of per-host static-threshold monitors reacting to restarts, deploys and routine spikes.
Changes, roughly in order of impact:
- Move up an abstraction level. Alert on the service or the SLO burn rate, not on 40 hosts individually. Per-host thresholds page a human for a node that self-heals.
- Make the condition harder to satisfy. Composite monitors requiring two signals (error rate high and latency high) cut false positives far more effectively than nudging thresholds.
- Tune the windows.
min_failure_duration, evaluation windows and renotify interval - a monitor evaluated over 5 minutes that renotifies every 10 is three pages for one incident. - Match the method to the metric. Seasonal traffic gets anomaly or forecast monitors instead of a fixed number; low-volume endpoints get a longer-window count rather than a percentage of three requests.
- Route, don't delete. Non-actionable signals go to a low-severity channel or a workflow that files a ticket. Keep the monitor, remove the page.
- Suppress planned change automatically - downtime at deploy time via Workflow Automation, not a human remembering to mute.
The target end state: pages are reserved for user-facing symptoms that have an owner and a runbook. Everything else is a ticket or a dashboard glance.
3. PagerDuty is going away next quarter. How do you land incidents and on-call in Datadog without turning the alert problem into an accountability problem?
Read the worked answer
- Ownership first. Populate the Service Catalog with the 40 teams, their services and their tiers. Without this, incidents land in a channel and stall, and on-call routing has nowhere to point.
- On-call. Build schedules per service and escalation policies that mirror tier: tier-1 pages immediately with a 5-minute acknowledgement, tier-3 goes to business hours. Keep the chain short - two tested levels beat a three-level chain nobody has ever exercised. Run a scheduled page test before PagerDuty is switched off, not after.
- Severity taxonomy. Define 3-4 severities with objective triggers (customer-visible outage, degraded, internal-only, security). Auto-declare incidents from monitors only for the top severity; everybody else declares from a slash command or the incident page.
- Automate the toil. A Workflow Automation on incident creation: create the Slack channel, add the on-call responder from the schedule, open the Jira ticket, post to the status page for SEV-1. This is what makes the migration feel like an upgrade rather than a lateral move.
- Close the loop. Post-incident review with a timeline drawn from the incident itself, action items as Jira tickets, and a follow-up check that the monitor which missed the incident has been fixed.
The judgement call: do not auto-declare incidents for every alert. Automation that manufactures 300 incidents a week converts a noise problem into an accountability problem, and once responders stop trusting the severity field the whole programme loses credibility. Ship the SLO burn-rate alerts first, then route incidents off the small set of signals that survived step two.