Eight Hundred Alerts and Five Nights in a Row
Solaris Freight runs 24/7 logistics operations on a platform shared by six product teams. Monitoring produces 800 alerts a week; 72% are closed with no action taken, and one engineer was paged on five consecutive nights last month. Three of the six teams have no runbooks, and only two roster anyone for out-of-hours support - the rest escalate to a central operations group of four, which also handles all deployments. Mean time to restore is 4.5 hours, of which roughly 60% is diagnosis. Two SLOs exist, both of the form 'service availability = up', measured by ping. Forty percent of incidents are change-related. Post-incident reviews happen, but only 30% of action items are ever closed. Traffic is growing 30% a year. Team A proposes hiring four more SREs; Team B proposes 'you build it, you run it' with 20% of each team's capacity budgeted for operational work. Leadership wants to scale without doubling headcount.
1. Evaluate both proposals. Under what conditions would each actually work?
Read the worked answer
Both are partly right and both are being sold as a silver bullet.
Team A - four more SREs. This treats a demand problem with supply. It works only if the four are chartered to remove toil through automation and self-service, measured in hours per week of toil eliminated. In practice, added central capacity absorbs the paging and entrenches the model where six teams own no operational responsibility. The economics also fail: 30% traffic growth outruns linear headcount, and 800 alerts a week is an alert design problem that more people will simply process more efficiently. Verdict: an acceptable temporary bridge with a hard toil-reduction target, not a strategy.
Team B - you build it, you run it, 20% capacity. The correct long-term direction, but conditional on teams being able to operate the service: usable telemetry, linked runbooks, safe deploys, and the authority to change the service to remove the cause of the page. Imposed without those, it simply distributes burnout across six teams. And 20% is realistic only after alert noise is cut; today that capacity would be consumed firefighting.
Practical synthesis: cut noise and build platform capability first, roughly three months, then transfer operational ownership team by team, starting with the team whose telemetry is cleanest. Keep a small central group for shared infrastructure and cross-team incidents, not for first response.
2. Redesign the SLIs, SLOs and alerting so that on-call becomes sustainable.
Read the worked answer
SLIs must measure what customers experience, not whether a process is alive.
- Per customer journey, not per service. Booking, tracking lookup, label generation, partner EDI intake. For each: availability (proportion of requests served successfully), latency (proportion under a threshold, for example booking p95 under 800ms), and correctness (proportion of bookings landing in the warehouse system without manual correction). In logistics, 'up but wrong' is the expensive failure, and ping-based availability cannot see it.
- Define 'good event' precisely. A 2xx response is not enough. A booking that returns 200 and is later corrected by hand is a bad event and must count against the SLI.
- Set SLOs with error budgets. For example 99.5% of booking requests succeed over 28 days, and 99% complete under 800ms. Budget burn drives paging, not raw thresholds.
- Alert on symptoms via multi-window burn rates. Page on fast burn (2% of the 28-day budget consumed in one hour) and slow burn (10% in six hours). Everything else becomes a ticket. Target fewer than 20 pages a week, each actionable, each linked to a runbook.
- Delete the noise rather than tune it. Most of the 72% non-actionable alerts should be removed outright; tuning preserves the habit of ignoring the pager.
- Write an error budget policy. When a team exhausts its budget, feature work pauses until reliability work ships. Without that consequence, SLOs are decoration.
Track pages per week, percentage actioned, alert-to-incident ratio, and budget consumed per team.
3. First 90 days: how do you fix the 30% action-item closure rate and the 60% diagnosis share of MTTR?
Read the worked answer
Action-item closure is a process design failure, not a discipline failure. Cap each review at three action items, each with one named owner, a due date, and a ticket in the backlog the team already uses - not a separate spreadsheet. Review them in the weekly team meeting and report the age of open items, not just closure percentage. Quality matters too: 'improve monitoring' never closes; rewrite items as changes with a verification test, such as 'add a canary stage with automated rollback, verified in a game day'. Track repeat incidents with the same root cause - that number, not closure rate, tells you whether learning is happening.
Diagnosis time is mostly an information-availability problem.
- One place with service dashboards, deploy markers on graphs, and trace IDs returned in error responses, so a responder moves from symptom to failing dependency without SSH.
- A runbook per alert, linked from the alert itself, starting with the three diagnostic commands. For the three teams with none, draft from the last two incidents rather than writing from scratch.
- An incident channel template capturing impact, timeline and current hypothesis, so handovers do not restart diagnosis.
- Game days for the top five incident types; the second occurrence should be measurably faster than the first.
- Measure time to first useful signal and time to identify cause, per incident, reviewed monthly.
Ordering: alert cleanup first, because it frees the capacity to do everything else; runbooks second; action-item cap third.