Repeat Major Incidents at a Managed Service Provider Under a Cost Cap

The figures describe a function that exists on paper and has no capacity, authority or queue discipline.

• 32% repeat rate (31 of 96). Almost a third of major incidents are known root causes. Analysis is being produced; remediation is not funded or owned. Check whether root-cause records end with an action assigned to a team that has no capacity - the classic orphaned action.
• Capacity. 96 major incidents a year is roughly two per week. A problem manager at 0.5 FTE, minus meetings and reporting, has perhaps ten investigation hours a week: one deep investigation per fortnight against two new incidents a week. The backlog can only grow, and two vacancies mean the method has restarted twice with no continuity.
• 62% emergency changes. Emergency change is being used as an incident workaround mechanism. It consumes the planned-change capacity that would prevent incidents, and emergency changes are themselves a known incident source - a loop that feeds the repeat rate.
• 58% first-time fix. Knowledge is not being captured from either incident resolution or problem investigation, so the same fixes are rediscovered by different people.
• CSI register: 74 open, 12 closed, no measured benefit. It is a list, not a managed pipeline. A register with no rejections has no prioritisation, and items without a benefit measure cannot be defended at funding time.
• 24% attrition. Every departure removes the tacit knowledge of why a workaround exists, raising repeat incidents independently of any process failure.

Conclusion: the binding constraint is capacity and ownership of remediation, not diagnostic skill. Buying tooling - already excluded - would automate a queue nobody works.

Constraint check first. On a 24x7 service, 99.5% means about 43.8 hours of unavailability a year and 99.9% means about 8.8 - you must remove roughly 35 hours. With 96 major incidents, the 31 repeats averaging three hours account for about 93 hours. Eliminating half the repeats removes about 45 hours, covering the target with margin. That points the money at recurrence, not at more monitoring.

Intervention 1 - recurring-defect elimination (fund first). A ring-fenced team of one problem manager plus two engineers, with authority to force a fix into a release. Cost is roughly three FTE. Payback case: SLA credits are USD 1.4M per quarter, about 5.6M a year; a 30% credit reduction pays for the team several times over. Target: repeat major incidents 31 → ≤10 in 12 months, tracked as mean time between recurrences.

Intervention 2 - change and certificate lifecycle standardisation. Automate certificate discovery and renewal, and require every emergency change to open a problem record. Target: emergency changes 62% → ≤25% and change-related incidents halved. The certificate case is the proof point: the control that would have prevented the second outage costs almost nothing.

Where the 12% run-cost reduction comes from. The same work: the top ten incident types typically drive 40%+ of 1,800 monthly tickets. Once the problem team has cleaned the categorisation, automate or self-service the top three.

Why not something else. An AIOps purchase is excluded and would fix neither root cause. Present this as a 90-day proof: baseline both measures in week one, report at day 90, release further funding only on evidence.

Design principles:

• Rename and separate. Call it a learning review and run it outside any disciplinary process, with a written guarantee that statements cannot be used in performance action. State the guarantee at the start of every review; said once in a policy document it is worthless.
• Trigger rules, not judgement. Mandatory for any major incident, any repeat of a known root cause, any outage over two hours, and any case where an on-call engineer was paged twice. Predictable triggers stop reviews becoming political.
• Timeline first, opinions second. Reconstruct from 30 minutes before detection to resolution using logs, chat and pager records. Then ask one question per decision point: what did the person know at that moment? Not: why didn't you check?
• Cap the outputs. Maximum three actions, each with a named owner, a date and a definition of done. Unlimited actions are how the CSI register reached 74 open items.
• Route actions into the problem backlog, not the CSI register, and report closure weekly to service delivery managers.
• Measure the process by outcome: repeat-incident rate and percentage of actions closed by the due date. Never report the number of reviews held - that rewards ritual.
• Feed knowledge articles from every review to attack the 58% first-time fix. Read the attrition figure as a design input: if the same fix is rediscovered after each departure, the review output should be an article, not a training session.

Leadership behaviour decides whether this survives. If a manager uses a review once to assign blame, engineers will reconstruct timelines defensively and the data quality collapses. Watch the first three reviews and intervene publicly if it happens.