The Holiday Hotfix Bypass
A SaaS company's release procedure requires two peer reviewers and change advisory board (CAB) sign-off for production changes. During a December freeze, a customer-facing billing defect needs a fix. The engineering manager approves a hotfix with one reviewer and no CAB, citing a 'critical incident' exemption in the procedure that requires the incident commander's written authorisation - which was not obtained. The fix deploys, and two days later a duplicate invoice issue affects 340 customers. The post-incident review shows the hotfix changed a tax calculation, and the second reviewer would likely have caught it. The engineering manager says the exemption was 'obvious' and the team was under pressure. The procedure also says emergency changes must be retrospectively reviewed within five business days; none has been scheduled. You are the process owner asked to decide whether to treat this as a disciplinary matter, a procedure failure, or both.
1. Assess the hotfix against the release procedure. Was this a justified emergency change, and what specifically failed?
Read the worked answer
The hotfix was urgent but not authorised. The procedure permits an emergency change only with the incident commander's written authorisation, and it requires retrospective review within five business days. Neither condition was met. The manager's belief that the exemption was 'obvious' is not authorisation, and pressure is not a substitute for the named approver.
Specific failures:
- Authorisation failure - no incident commander sign-off and no record of who declared a critical incident.
- Control failure - one reviewer instead of two, and the missing second review would likely have caught the tax calculation change.
- Record and follow-up failure - no retrospective review was scheduled, so the organisation did not learn from the change inside its own window.
- System failure - the deployment pipeline allowed a production change without the required approval token, so the control depended on memory and goodwill.
This is both an individual and a system issue. The manager chose to bypass the authorisation step; that is an accountability question. The system also made the bypass easy, provided no automatic gate, and did not create a retrospective review ticket. Treating it only as discipline would miss the control weakness; treating it only as a process failure would excuse an intentional deviation. The 340 affected customers make the consequence material, not trivial.
2. Recommend corrective actions that address both the individual decision and the system. Prioritise them.
Read the worked answer
Prioritise by harm and repeat likelihood.
Immediate, this week:
- Schedule the retrospective review within the five-day window and freeze further emergency changes in the billing domain until it is complete.
- Contain the duplicate invoices: identify all 340 customers, correct billing, issue credits, and notify finance and customer success.
- Preserve the change record and deployment logs for the review.
Individual accountability:
- Refer the manager's decision to the appropriate HR or performance process, focused on failure to obtain the named authorisation, not on the urgency itself.
- Require coaching on the emergency change procedure and on when an incident commander must be involved.
System fixes:
- Make the gate system-enforced: the deployment pipeline requires an incident commander approval token for emergency changes.
- Add a second-reviewer requirement for billing and tax code, with an automated path check.
- Auto-create a retrospective review ticket with a five-day due date and an owner.
- Clarify the procedure: define who can declare a critical incident, who the named authorisers are, and what 'written' means.
- Add a pre-approved emergency template for known billing fixes to reduce pressure to improvise.
Avoid making 'retrain everyone' the main action. Training does not fix a pipeline that permits bypass. Track emergency changes per month, second-reviewer compliance, and retrospective review completion.
3. How should the retrospective review be conducted and what should be communicated to affected customers? Consider the process owner's role and the engineering manager's continuing authority.
Read the worked answer
The retrospective review should be independent of the manager who approved the hotfix. Include engineering, QA, finance/billing, and the process owner. Examine what changed in the tax calculation, why the second reviewer would have caught it, whether other tax cases are affected, and whether rollback or repair is complete. Document findings and actions in the change record. The review is blameless about system causes but not blind to the missing authorisation; it should state both.
Customer communication should be factual and timely: apologise for the duplicate invoices, explain the correction and credit, give a contact route, and avoid blaming an individual. Coordinate with legal and customer success, and keep the message consistent with what the regulator or payment partners may see.
As process owner, you own the review and the corrective actions, not the disciplinary decision. Consider a temporary control: for 90 days, emergency changes in billing require co-approval by a second named manager. That is proportionate and protects the service while the system fixes land. Do not leave the manager as sole approver in the same domain without a check.
Internally, separate 'what happened' from 'who is accountable'. The phrase 'the exemption was obvious' is a normalisation-of-deviance signal: the team believed informal judgement was acceptable. Address that with a clear rule and a safe route to escalate pressure, while preserving individual accountability for an intentional bypass.