A Quarterly Report That Would Overstate Detection
You are drafting the quarterly security operations report for a 3,000-employee insurer. The raw numbers: 412,000 alerts, of which 380,000 came from a single signature; automation closed 96% within two minutes; 41 incident tickets were opened; reported mean time to detect is 47 minutes; the SOC's stated ATT&CK coverage is 98%, computed from techniques that have a rule mapped to them, most of which have never fired. In the same quarter, a ransomware precursor was found only when a customer phoned the helpdesk about a phishing email - nine days after delivery. The SOC lead wants the report to show improvement over last quarter's three-hour MTTD. The CISO wants the board to see strong detection, and next year's budget depends on the narrative.
1. Critique the MTTD and coverage figures and say what you would replace them with.
Read the worked answer
Both figures measure the SOC's own plumbing, not the organisation's exposure.
MTTD of 47 minutes is almost certainly measured from alert creation to triage, which makes it a measure of queue latency, not detection. The ransomware case demonstrates the gap: the real detection time was nine days, and the detector was a customer. The honest figure for that incident is dwell time from first observable - the email delivery timestamp - to case creation. A mean over 41 tickets that excludes the one incident that mattered is worse than no metric, because it invites the board to conclude they are well defended.
98% coverage counts rules that exist, not rules that work. A technique with a mapped rule that has never fired is untested, not covered. Worse, the rule set is dominated by one signature producing 92% of all alerts, which means analyst attention is being spent on a single noisy condition.
Replacements:
- Detection efficacy measured by adversary emulation: percentage of priority techniques that produced an alert in a purple-team test in the last 12 months, with the misses named.
- Median and 90th-percentile time from first observable to case creation, computed from raw telemetry, not ticket timestamps.
- Alert-to-case ratio per rule, plus a named list of the top five rules by volume and their true-positive rates.
- Time from first report to containment for externally reported incidents.
Report the nine-day case as the headline data point. It is the most informative number in the set.
2. Write the three-to-four sentence key message for the board that is honest without being self-flagellating.
Read the worked answer
A defensible example:
'Our monitoring produced 412,000 alerts this quarter and our team resolved 96% of them within two minutes, which is fast but tells you more about automation than about detection quality. One ransomware precursor reached us only through a customer call nine days after the phishing email arrived, and that is the number I want the board to remember: nine days, not 47 minutes. We have therefore stopped reporting average detection time based on alert timestamps and will report time-to-detect from the first observable signal, alongside independently tested detection coverage. The budget request funds the two things that would have shortened those nine days: phishing-resistant MFA on the mail path and adversary-emulation testing of our top detection rules.'
The structure is: state the operational strength factually, put the failure next to it in plain language, explain the measurement change so it reads as rigour rather than retreat, and tie the ask to the specific failure. Do not use the words 'mature', 'robust' or 'world-class' anywhere in the document.
3. The SOC lead pushes back, saying the nine-day case was an outlier and should be footnoted rather than led with. How do you handle that, and what evidence would change your position?
Read the worked answer
Treat it as a measurement-design disagreement rather than a loyalty test, and answer it on method.
The argument for leading with it: it is the only case in the quarter where an adversary achieved something. The other 40 tickets describe events the SOC contained. A metric that excludes the sole successful intrusion is not an average, it is a selection. Boards make risk decisions from tail events, and a footnoted tail is a hidden tail.
Where the lead could be right: if the nine-day gap was caused by a control outside SOC scope - for example, the mail gateway was not licensed to detonate attachments, and that gap has since been closed - then the case is a procurement finding, not a detection finding, and the report should say so explicitly. That is a legitimate reframe and I would accept it with evidence.
Evidence that would change my position:
- Proof the case was already in the queue and mis-triaged - that makes it an analyst performance issue, reportable separately.
- Proof the delay came from a vendor outage or a documented, accepted risk decision.
- A larger sample showing externally reported incidents are the norm rather than a single event - which would strengthen rather than weaken the case for leading with it.
Either way the number stays in the document with its measurement basis stated. What is negotiable is the framing sentence, not the inclusion.