The 71% SLA Breach That Shouldn't Be Possible
Northwind Health runs Security Incident Response on a single domain. Thirty-eight analysts sit in three assignment groups: Tier1-SOC (28), Endpoint (6), and Network (4). Coverage is follow-the-sun: APAC 22:00-06:00 UTC, EMEA 06:00-14:00, AMER 14:00-22:00. A SIEM integration creates roughly 900 security incidents per day. Every incident uses one SLA definition with a 4-hour response target, attached to the 8x5 'Business Hours' schedule. Six weeks after go-live, 71% of P2 incidents are reported as breaching the response SLA, yet the median time from creation to the first analyst work note is 11 minutes. Assignment is a single round-robin to Tier1-SOC regardless of category; Tier1 manually forwards endpoint incidents after triage, and that handoff has a median of 3 hours. Endpoint currently holds 5,900 open incidents; Network holds 300.
1. The 71% breach rate looks arithmetically impossible given an 8x5 schedule and an 11-minute median first comment. Enumerate the plausible explanations and give the order in which you would check them.
Read the worked answer
The contradiction is the clue: either the clock is not the one you think it is, or the report is not counting the thing you think it is.
Check 1 - what does 'breach' actually count? Pull 20 incidents flagged as breached and compare the SLA record's start, pause, and stop timestamps against the incident's own activity timestamps. This is cheap and eliminates half the hypotheses immediately.
Check 2 - is the schedule really applied? An empty schedule field, a schedule with no entries in the analyst timezone, or a schedule in the wrong timezone all produce a 24x7 clock. Symptom: breach rate clustered by hour of creation, with incidents created 22:00-06:00 UTC breaching far more. Bucket the breach rate by creation hour; if it is flat, the schedule is not doing anything.
Check 3 - what stops the clock? If the stop condition is a state change or a field nobody sets on first contact, the 11-minute comment is irrelevant. Read the pause and stop conditions and ask what an analyst literally does in the first ten minutes.
Check 4 - multiple SLA rows. Reassignments or state transitions can create additional SLA records per incident. If the report reads SLA rows rather than incidents, it double-counts, and response tasks with their own targets inflate the rate further.
Change configuration only after 1-4, otherwise you fix the schedule and still breach.
2. Quantify the capacity problem and propose a routing design. What would you change first, and what regression would you watch for?
Read the worked answer
Load math. Endpoint: 5,900 open across 6 analysts, roughly 983 per analyst. Network: 300 across 4, roughly 75 per analyst. The imbalance is not a routing bug, it is a structural one - a single round-robin queue with a 3-hour manual handoff in front of the only group that can resolve most of the volume.
Design.
- Route at creation, not after triage. That requires the SIEM integration to populate category/subcategory reliably; if it does not, assignment rules cannot fire and you get the round-robin default. Verify field population on the integration's source records before writing rules.
- Rules keyed on category and, where it exists, on a CI-based lookup to the supporting group - not on analyst name.
- Keep Tier1 as owner by default and escalate on an explicit condition (confirmed endpoint malware, EDR containment available) rather than a manual judgement call.
- Round-robin distributes volume, not skill or capacity. Prefer assignment groups sized against measured arrival rate.
Change first: the 3-hour handoff. It is pure queue time and it is invisible in your current SLA if the clock stops on assignment. Instrument 'time in Tier1 queue' as its own metric.
Regression to watch: auto-assignment silently overwriting manual reassignment, and rules that fire on every update rather than on create, which reassigns an incident mid-investigation. Also watch the second-order effect: once endpoint routing is automatic, Tier1 loses visibility of endpoint incidents entirely and coverage gaps get quieter, not smaller.
3. You inherit 5,900 open Endpoint incidents this week. What is your immediate plan, and what is the risk in it?
Read the worked answer
Do not bulk-close blind, and do not work the queue in created-date order either.
- Profile before touching. Group the backlog by detection signature and CI. In most SIEM-fed queues a small number of signatures produce the majority of records; the same host appearing repeatedly means either a persistent compromise or a noisy rule. Ten minutes of grouping tells you which problem you have.
- Collapse duplicates. Where the same CI, signature, and a short time window repeat, keep one incident and close the rest as duplicates with a reason code, linked to the survivor. Auditability is preserved because the closed records still exist and are still linked.
- Sample the stale tail. Take 30 of the oldest and 30 of the most common signature and actually look. If they are blocked-by-EDR commodity detections, closing them as informational is defensible. If any show lateral movement, your backlog is an active incident, not a backlog.
- Fix the source. Every hour spent on the queue without a ticket to the SIEM rule owner is an hour you will spend again next month.
Risk: closing records destroys the only evidence that a host was repeatedly flagged, and 'closed as duplicate' can mask a genuine second-stage compromise on the same host. Mitigate by keeping the original record, the reason code, and a named analyst on any bulk operation - and by sampling, not assuming.