Breaking a quarter-end change freeze for a critical authentication vulnerability
A European retail bank is two days into a 10-day change freeze protecting its quarter-end financial close. All production change is barred except emergency change. That morning the vendor of the online banking authentication library discloses a CVSS 9.8 vulnerability with a public proof-of-concept exploit. The fix is a library upgrade requiring redeployment of the authentication service and an additive database column. The service handles about 200,000 logins a day and is internet-facing. The vendor has run the patch in its own production for five days with no reported issues. The bank's emergency change process requires an emergency change authority decision within four hours and a documented 30-minute rollback; the redeploy itself takes 45 minutes. The risk committee asks the release manager whether to break the freeze, and if so, how.
1. Should the freeze be broken? Justify the decision in risk terms rather than by procedure.
Read the worked answer
Yes - but as a specific, time-boxed emergency change, not as a general suspension of the freeze.
The freeze exists to stop unplanned change risk landing on financial close. The comparison is not 'change versus no change' but 'this change versus an unpatched, internet-facing, public-exploit authentication service for at least eight more days'. An exploited authentication bypass would threaten the close itself, plus customer funds and a regulatory notification. The probability may be moderate and hard to estimate, but the impact is severe and the exposure window is long.
The change risk looks contained: narrow scope (one library plus an additive, backward-compatible column), a vendor build already running in the vendor's production for five days, and a service that is well understood.
What makes this defensible is framing: a named emergency change authority makes a recorded decision, the deployment is scheduled in a low-traffic window, and interim compensating controls - tighter WAF rules, rate limiting on authentication endpoints, enhanced monitoring for anomalous login patterns - start immediately rather than waiting for the patch.
2. The process requires a documented 30-minute rollback, but the redeploy takes 45 minutes. How should the release manager handle this?
Read the worked answer
Do not fabricate a 30-minute rollback. Either engineer a real one or escalate a documented deviation.
Engineering options, in order of preference:
- Blue-green on the authentication service. If both versions can run simultaneously, rollback is a traffic switch, typically well under a minute. This is the cleanest answer and worth the extra preparation even if it pushes the change by a few hours.
- Forward-only framing for the database. The new column is additive, so the previous library version should still run against the new schema. That means no database rollback is needed at all - only an application version switch. Confirm this rather than assuming it.
If neither is achievable in the window, the release manager escalates to the emergency change authority and proposes accepting a deviation: rollback by redeploy in 45 minutes, with the vendor on the bridge, a validated pre-built artifact ready to go, and recovery-time objectives agreed in writing.
The point is that the 30-minute figure is a control expressing 'we must recover quickly'. What matters is that the true recovery time is known, tested, and accepted by someone with the authority to accept it.
3. What should happen after the incident closes, in terms of both records and policy?
Read the worked answer
Records: the emergency change should be closed with a full release record - artifact version, library version, deployment timestamps, approver, rollback method and the deviation accepted. Update the configuration management data for the authentication service and the definitive media library entry for the released build. Add the vulnerable version to the known-error or advisory record so other services using the same library are found.
Verification: check logs for exploitation attempts during the exposure window; if any are found, this becomes a security incident, not just a change.
Policy: this event exposes a gap - a freeze with no pre-agreed category for security-critical change invites either a breach of the freeze or a delay that is indefensible. Propose a standing exception category: changes remediating a vulnerability above an agreed severity threshold, with a defined approver, an evidence requirement, and a shortened but real rollback plan. That converts an ad-hoc judgement under pressure into a rehearsed one, while keeping the freeze meaningful for ordinary functional change.