Meridian Telco: Enabling Auto-Remediation on 4,200 Cell-Site Routers
Meridian Telco operates 4,200 cell-site routers. A flapping backhaul link can generate 300 alarms in ten minutes. The AIOps engine already correlates these into a single incident and recommends 'restart router'. Correlation accuracy on last year's incidents is good, and the vendor now proposes enabling auto-execute whenever model confidence exceeds 0.90. Last quarter a partner ISP running the same feature rebooted a core router during peak hours; the resulting 40-minute regional outage affected 210,000 subscribers. Meridian's confidence scores have never been calibrated and are known to be overconfident on rare alarm signatures. The change board wants auto-execution live before the next fiscal quarter, and the regulator requires that any automated action affecting subscriber service be attributable to a named responsible engineer.
1. Design the autonomy rollout. What must be true before any action executes without a human, and what hard limits apply at full autonomy?
Read the worked answer
Graduated autonomy with explicit gates, not a single confidence threshold.
Stage 0, four to eight weeks: shadow mode. Recommendations are logged but never executed, and engineers record whether they would have approved each one. This produces a false-remediation rate and a near-miss rate on the real distribution, including rare alarm signatures.
Stage 1: execute-with-approval, restricted to single non-core routers with verified redundancy and off-peak windows.
Stage 2, auto-execute only when all of the following hold: scores are calibrated (reliability curve checked, isotonic or Platt scaling applied) so 0.90 means 0.90; blast-radius limits of at most three devices per ten minutes and never a router carrying more than a set share of regional traffic; no action during change freezes or declared incidents; automatic post-action health check with rollback after five minutes; and a kill switch any engineer can pull.
The partner ISP outage was not a modelling error. Restart is often the right action and still catastrophic at the wrong blast radius. Cap scope first, then relax.
2. The change board asks who is accountable when an automated restart causes an outage. Answer them, and specify what evidence the system must retain.
Read the worked answer
A named human engineer remains accountable, and that is not a formality. Someone chose to enable autonomy, set the threshold, defined the blast radius, and approved the policy that let the machine act. 'The model decided' is an abnegation of duty and will not satisfy a regulator who requires attribution.
Therefore: each autonomy policy carries an owner, a version, and a named approver, reviewed quarterly. Each executed action stores the model version, the calibrated score, the input signals that drove it, the policy version in force, the resulting device state, and the identity of the policy owner. The on-call engineer needs a readable explanation at 3 a.m. - top contributing alarm signatures and the two or three most similar historical cases with their outcomes - not a feature-importance vector.
Also handle subscriber data: alarm payloads can contain identifiers, so retention and access for the audit trail must be scoped to what an investigation actually needs. And keep the model's remit strictly technical; it must never become an input to individual engineer performance review, or incident reporting will degrade immediately.
3. What single measurement would you require before signing off full auto-execution, and how would you obtain it?
Read the worked answer
The false-remediation rate: the share of executed actions that made service worse or failed to improve it, measured against a counterfactual.
You cannot get it from production without executing, so obtain it by replaying twelve months of historical incidents through the current engine and scoring every recommendation against what actually happened afterwards, supplemented by the shadow-mode log where engineers recorded accept/reject and outcome. Segment by alarm signature, because aggregate accuracy hides the rare classes where overconfidence lives.
Set the bar in advance and in business terms: for example, fewer than one in 500 executed actions causing measurable degradation, and zero executions on core routers in the replay. If the replay cannot produce enough core-router cases to estimate the rate, that absence is itself the answer - the evidence does not exist, so the autonomy boundary stays where the evidence supports it.