Automating the Change Advisory Board
Every Friday, Ferrolane Retail releases 900 changes into production; 82% are classed standard. Eighteen months ago the change manager introduced a model that scores each change for risk and auto-approves anything below a threshold, removing it from the change advisory board agenda. The board shrank from 14 people to 4 and lead time for low-risk change fell from 9 days to 6 hours. In the last six weeks two Sev-1 outages traced back to auto-approved changes: a pricing-rule change that duplicated discounts, and a database parameter change that broke replication for 11 hours. The model was trained on three years of board decisions, which the change manager now concedes were largely rubber-stamped after the first five minutes. The internal auditor found no record explaining why any individual change was auto-approved, only a score. The CIO wants the board restored to full review of every change.
1. Diagnose why the model's risk scoring failed to prevent these two outages. Address the training data, the control design and the value stream.
Read the worked answer
Training on past board approvals teaches the model what the board did, not what was risky. If the board approved almost everything it saw, the model's ceiling is "this resembles things we approved". Two structural faults follow: label leakage, where the model learns features correlated with approval - submitter, change window, team - rather than with failure; and the absence of an outcome label, because nobody labelled changes by whether they later caused an incident. It is a decision-replication model, not a risk model. Both outages almost certainly scored low because they looked entirely ordinary.
Control design: a threshold with no per-change explanation is a control that cannot be audited, challenged or learned from. The auditor's finding is the real failure - the value of the model is not the score but the evidence trail. Auto-approval also removed a control without adding detection: replication breakage should have been caught by a post-change health check or canary, not by a change record. The 82% "standard" classification is doing unexamined work too; if standard changes are exempt from review by definition, the classifier assigning that class becomes the most powerful control in the pipeline.
Value stream: the constraint moved rather than disappeared. Lead time fell, but failures were absorbed downstream at 03:00 by operations, which is where the true cost now sits. Look at where change is tested and verified, not only where it is approved.
2. The CIO wants the full board restored. Give your recommendation with reasoning, and specify what evidence the pipeline must record per change.
Read the worked answer
Restoring a 14-person board over 900 changes recreates the rubber stamp and adds roughly three days of lead time to prevent a handful of failures - a poor trade that would be reversed within a quarter. Recommend tiered assurance instead, aligned to the score but not decided by it: auto-approval only where an automated technical control verifies the outcome (canary, health checks, auto-rollback) and only for change types with a measured failure history; medium score goes to a named reviewer with a two-hour response commitment, not a meeting; high score keeps board review. Rebuild the model with outcome labels - change linked to incident within 14 days - and validate on a held-out time window rather than a random split, because the purpose is predicting future failure, not explaining the past.
Evidence required per change: model version and the threshold in force at the time, the top contributing factors with their values, the identity of the human who set that threshold and when, the automated verification results, and the decision outcome. That gives an auditor a reconstructable rationale rather than a number.
Add a monitoring loop: every auto-approved change that produced an incident is reviewed monthly by the change manager and product owner as a formal improvement input, and the threshold is re-justified at the same meeting. Retire the automatic standard-change exemption - classification should route assurance, never remove it.