Emergency Pricing Fix During Peak Freeze
Northwind Retail runs a pricing service that calculates loyalty discounts at checkout. A change freeze is in effect from 15 November to 2 December to protect Black Friday trading; 47 production changes are blocked. On 22 November, the service starts applying a 20% discount instead of 5% to about 3.4% of carts, costing an estimated $85,000 per hour. The cause is a one-line feature-flag configuration error deployed on 14 November. A revert has passed staging tests and takes five minutes, with a documented rollback. The change policy requires emergency changes to be approved by the CAB chair, the service owner, and security. The CAB chair and service owner are unreachable for two hours; security will approve only if an untested edge case involving gift cards is accepted as a known risk. The operations manager wants to bypass the freeze immediately.
1. Classify this change and decide whether the freeze should be broken. Justify the approval path and evidence you would require.
Read the worked answer
This is an emergency change, not a standard change, because the configuration revert is not pre-authorised in the standard change catalogue. The freeze is a risk-control mechanism, not an absolute prohibition; breaking it is justified only when the cost of waiting exceeds the risk of changing. Here the estimated loss is $85,000 per hour and the technical change is a one-line revert with a five-minute rollback, so the balance favours action.
The approval path should still be followed. If the CAB chair and service owner are unreachable, use documented delegated authority rather than bypassing the process. Security's condition should be handled explicitly: accept the gift-card edge case as a recorded risk, test it in a canary slice, and define the fallback if it fails. Require the service owner's delegate, security approval, and a named risk acceptor. Record the decision, the evidence, and the rollback trigger in the emergency change record, with retrospective CAB review within 24 hours.
The key judgement is that speed does not remove governance. A bypass would set a precedent and leave no audit trail.
2. What deployment and post-implementation steps would you mandate?
Read the worked answer
Deploy through a controlled canary rather than to all traffic at once. Send 5% of checkout traffic through the reverted configuration, monitor discount accuracy, error rate, checkout conversion, and gift-card behaviour for a defined observation period, then expand if the metrics are clean. Keep the feature flag capable of immediate rollback and have the on-call engineer ready to trigger it.
Before deployment, confirm the exact configuration diff and that staging evidence covers the affected discount paths. After deployment, verify sample carts for each loyalty tier and reconcile a sample of orders against expected pricing. Communicate to customer service and finance so they can handle refunds or adjustments.
Post-implementation, hold a review within 24 hours covering why the error escaped testing, whether the freeze exception was correctly used, and what controls should change. Likely actions include an automated test for discount configuration, a pre-approved standard change for low-risk config reverts, and tighter feature-flag approval. Update the emergency change record with actual timings, impact, and risk-acceptance evidence.