Change Management By-Pass in a High-Velocity CI/CD Pipeline
Meridian Insurance deploys about 60 times a week through a CI/CD pipeline. Twenty-two engineers hold permission to push directly to production, and the change advisory board reviews only changes estimated at more than 40 hours of effort; everything else is classified 'standard.' Internal audit sampled 50 production changes and found 14 with no evidence of testing and 3 that could not be tied to any ticket. Two engineers can both approve and deploy, and hold standing production access. One unticketed change altered the formula that calculates statutory interest on late-paid claims, raising it from 8% to 9.5% for roughly 11,000 open claims before anyone noticed 19 days later. The CTO argues that slowing releases would cost the company its market position.
1. Evaluate the design of the change management control, including the 40-hour threshold. Is this an operating deficiency, a design deficiency, or both?
Read the worked answer
Both, and the design problem is the more serious one.
The threshold is not a risk criterion. Effort hours measure how hard a change was to build, not what it can break. The change that moved statutory interest from 8% to 9.5% was almost certainly a few lines - well under 40 hours, therefore 'standard,' therefore unreviewed. Meanwhile a 60-hour change to a reporting label would have gone to the CAB. The control is designed around the wrong variable; it needs to be tiered by blast radius (financial calculation logic, authentication, data model, customer-facing pricing) with size as a secondary factor.
'Standard change' is being used as a residual bucket. A standard change category is only valid for pre-approved, repeatable, low-risk types with a defined template - a password reset job, a config change to a known parameter. Here it means 'anything under 40 hours,' which is the entire population of changes that actually carry logic risk.
Segregation of duties is absent by design. Two engineers who can approve and deploy, with standing production access, means the approval step is advisory. This is a design flaw: adding a reviewer to a process where the same person can act alone does not create a control.
Operating evidence confirms the design is not working. 28% of sampled changes lacked testing evidence and 6% could not be tied to a ticket. That is not a few lapses; it means the control cannot be relied upon for the population - you must assume the untested population is uncontrolled, not that 72% compliance is acceptable.
What I would tell the CTO: none of this requires slowing releases. It requires routing changes through automated gates by risk tier. High-velocity shops with strong controls typically deploy more often, because unreviewed changes are what cause the outages that eat release capacity. The 19-day undetected interest change is the cost of the current model, and it is measurable.
2. Analyse the interest formula change. What is the exposure, what would you do about the 11,000 open claims, and what does the root cause tell you?
Read the worked answer
Quantify before you opine. 1.5 percentage points on late-paid claims across roughly 11,000 open claims, for 19 days, plus every claim paid under the wrong formula since the change went live. The exposure depends on average claim value and days outstanding - compute it, and compute it in both directions:
- If the statute sets 8% as a fixed rate, Meridian has overpaid claimants (a recoverable or non-recoverable loss, depending on contract terms) and understated its own position.
- If the statute sets 8% as a minimum, Meridian has paid correctly but has been operating outside its approved configuration, and the 9.5% figure may have no documented basis.
Both are potentially material in different ways: one is a financial misstatement and a recovery question, the other is a compliance and disclosure question.
Actions
- Identify the author, the intent, and the approval trail - or its absence. Determine whether this was a well-intentioned change made under time pressure or something else. That changes the tone of the response and the HR implications.
- Recalculate the affected claims from source data, not from the output of the modified formula. Reconstruct the correct figure independently.
- Correct open claims before payment; determine the treatment of claims already paid (adjust, recover, or absorb) with counsel and finance.
- Determine whether the misstatement is material to the period; if it is, it goes to the auditor with a proposed adjustment.
- Assess regulatory reporting obligations - statutory interest on insurance claims is a regulated calculation in most jurisdictions, and a regulator may need to be told.
- Notify customers who were paid incorrectly, with a plain explanation.
Root cause. Not 'an engineer made a mistake.' The root cause is that changes to financial calculation logic could be made and deployed without independent review, without a ticket, and without detection for 19 days - and were detected by a customer complaint or a staff observation rather than by a control. The fix is a control applied to the class of change, not discipline applied to the individual. It is also worth noting that 19 days of undetected error in a statutory calculation implies no reconciliation or monitoring of claims payment outputs - a detective control gap that is broader than change management.
3. Redesign the control environment so that it survives contact with 60 deployments a week. Address SoD, emergency changes, and how you would demonstrate to the CTO that control and velocity are not opposites.
Read the worked answer
Tier changes by blast radius, not by effort.
- Tier 1 - financial calculation logic, authentication and authorisation, data model, pricing, regulatory outputs. Mandatory independent code review, automated test coverage gate, and a named business owner sign-off. The interest formula lives here.
- Tier 2 - customer-facing functionality and integrations. Automated testing plus post-deployment monitoring and a rollback plan.
- Tier 3 - cosmetic, copy, configuration within approved bounds. Pre-approved standard changes with automated checks only.
Classification should be enforceable in the pipeline (path-based policy-as-code), not left to the developer's judgment - otherwise the tiering inherits the same flaw as the 40-hour rule.
Segregation of duties, fixed structurally. Remove standing production access; require just-in-time elevation with a business justification and an expiry. Enforce approver cannot equal deployer in the pipeline itself, so the control does not depend on people remembering. The two engineers who currently hold both rights are the first fix, and it is a permissions change, not a project.
Emergency changes, defined properly. Break-glass exists because production fires are real. Define it: a documented reason at the time, a second person notified, automatic retrospective review within 24–48 hours by someone independent, and a monthly report of every break-glass use to the CTO. Emergency is a category with an audit trail, not an exemption from one.
Detective layer, which is what was actually missing. Automated reconciliation of claims payment outputs against expected parameters; alerting on changes to calculation logic regardless of tier; deployment frequency and change-failure-rate dashboards reviewed weekly. The interest error survived 19 days because nothing was watching outputs.
Answering the CTO on velocity. Measure it. Track deployment frequency, lead time, change failure rate, and mean time to restore. If the current model is genuinely fast, the metrics will show it - and they will also show the cost of the 14 untested changes and the 19-day error. The argument to make is not 'slow down'; it is that the incident cost more release capacity than the review gate would have. Offer a 90-day pilot on Tier 1 changes only, with the metrics reported either way. That is a testable proposal, and it moves the conversation from principle to evidence.