Black Friday: Incident, Known Error and a No-Blame Review
On Black Friday, an online retailer's payment gateway times out for 35 minutes. The incident manager fails over to a secondary gateway, and service is restored. During the review, problem management finds that the same failure occurred four months earlier and was logged as a known error. A fix was proposed, but change enablement rejected it because the business case showed no cost saving and the change freeze was three weeks away. The post-incident review names the engineer who applied the last gateway configuration as 'the cause' and closes with an action for 'more careful work.' The retailer lost an estimated £1.2 million in sales. The CTO wants a full incident review, a new change freeze policy, and a 'no-blame' statement, but staff say the last no-blame review still resulted in a formal warning.
1. Was the incident manager's failover the correct action? Distinguish incident management from problem management in your answer.
Read the worked answer
Yes. Incident management's purpose is to restore normal service as quickly as possible and minimise impact. Failing over to a secondary gateway within 35 minutes is a valid workaround, and the incident manager should not delay restoration to investigate root cause. Problem management has a different purpose: reduce the likelihood and impact of incidents by finding causes and managing known errors and workarounds. The failure here is that the same event occurred four months earlier and was logged as a known error, but the known error did not lead to an effective fix or a tested workaround. The incident manager's action was correct; the system around the known error was not. The review should separate the restoration decision from the problem management failure. Naming the engineer confuses the two: the engineer applied a configuration, but the known error and change decision were organisational. The £1.2 million loss belongs to the problem and change story, not the failover.
2. Evaluate change enablement's decision to reject the fix because it showed no cost saving and a change freeze was near.
Read the worked answer
Change enablement should maximise successful changes by assessing risk and authorising changes properly. Cost saving is not the only or main criterion; risk reduction is a legitimate reason to change. The known error had already caused one outage, and the proposed fix would reduce the likelihood of a repeat. Rejecting it because there was no cost saving ignored the risk exposure. The approaching change freeze was also handled poorly: change freezes should have an exception process for changes that reduce risk, especially before a peak trading period. The fix could have been assessed as a normal change with a risk-based change authority, or as an emergency change if timing was tight. The practice should have connected problem management's known error to change enablement, with an impact assessment and a clear decision record. A blanket freeze without risk exceptions is not control; it is avoidance.
3. Design a post-incident review and follow-up that uses continual improvement and organizational change management. What should the CTO do first?
Read the worked answer
The CTO should first make the no-blame commitment credible, because staff have seen a previous no-blame review end in a formal warning. That means separating accountability for deliberate misconduct from learning about system conditions, and stating that the engineer will not be disciplined for the configuration action. Then use the continual improvement model: where are we now - build a timeline with incident, problem and change records; where do we want to be - define the outcome as no repeat payment outage and a faster, safer change path; how do we get there - actions on known-error ownership, change freeze exceptions, gateway testing and on-call decision rights; take action; did we get there - measure repeat incidents and change lead time; keep momentum - publish progress.
Organizational change management should identify stakeholders (engineers, change board, problem managers, finance), communicate why the review is different, train change authorities, and address resistance. The CTO must resource the actions and follow through visibly, or the no-blame statement will not be believed.