Multi-supplier authorization failures at month-end
A retail bank's month-end payment processing is failing: 35% of card authorizations are rejected. The bank uses a cloud IaaS provider, a third-party payment gateway, and an outsourced service desk. The cloud status page reports all regions healthy. The gateway vendor says its logs show elevated latency from the bank's cloud environment and blames the cloud provider. The cloud provider says its network metrics are within contract and blames the gateway. The outsourced service desk is following its own major incident script and has not invited either vendor to a joint bridge. Each contract has a 99.9% availability SLA with credits, but none defines end-to-end payment authorization. The bank must notify its regulator within four hours of a material operational incident. The incident manager wants to fail traffic to the secondary region, but that requires an emergency change and a 30-minute DNS cutover with rollback risk. The CTO wants 'all vendors on one call now'.
1. As the bank's incident manager, how would you coordinate this multi-supplier major incident? Address decision rights, information sharing, and the regulator.
Read the worked answer
I would establish a single end-to-end major incident with one incident manager who has authority across the bank and its suppliers. The outsourced service desk's script is component-level; it must not run the end-to-end response. I would convene a joint bridge with named duty managers from the cloud provider, the gateway, and the outsourced desk, plus the bank's service owner, technical leads, communications, and compliance. Each supplier owns diagnosis and recovery within its domain; the incident manager owns coordination, escalation, timeline, and decisions. The CTO's request is right in principle, but 'all vendors on one call' needs structure: one chair, one status format, one decision log.
Information sharing must be factual and time-stamped. Each supplier should provide telemetry, not blame: latency percentiles, error rates, regional routing, and change history. I would agree a shared severity definition and a common impact statement - 35% authorization rejection at month-end - and ask each supplier to state what it can see and what it cannot.
The regulator notification clock starts when the incident is assessed as material. Compliance should be engaged immediately, not after technical resolution, so the four-hour deadline is met with accurate facts. Legal privilege and regulatory wording should be handled by compliance, while the incident manager supplies the timeline and impact data.
2. Evaluate the proposal to fail traffic to the secondary region. What should incident management, change enablement, and business continuity decide, and in what sequence?
Read the worked answer
Failover is a restoration action, but it is also a change with real risk: a 30-minute DNS cutover, possible data replication lag, and a rollback that may not be clean. It should be evaluated as an emergency change, not as an informal technical experiment.
The sequence should be: first, confirm the failure is likely to be regional and that the secondary region is healthy and can handle month-end volume; second, confirm the rollback criteria and who can authorize them; third, get emergency change approval from the delegated change authority, with the CTO and service owner aligned; fourth, execute with the cloud and gateway suppliers on the bridge, monitoring authorization success rates; fifth, validate recovery against business transactions, not just network health; sixth, communicate to customers and the regulator as required.
Business continuity should be involved because a failed cutover could turn a degradation into a full outage. If the failover is likely to restore service faster than continued diagnosis, the risk is justified. If replication is unverified, a staged approach may be safer. Incident management should keep the problem investigation running in parallel, but restoration takes priority. After recovery, preserve logs and a decision timeline for the post-incident review and any regulatory follow-up.
3. What contract and governance changes should follow this incident? Justify them in terms of the wider service value system rather than supplier blame.
Read the worked answer
The incident exposes a value-system design flaw: each supplier is measured on its own component SLA, but the bank's customers experience an end-to-end payment service. The contracts should define an end-to-end authorization outcome, shared service-level objectives, joint major incident obligations, out-of-hours escalation, information-sharing rights, and participation in post-incident reviews. Supplier credits should not reward a supplier for being 'within contract' while the end-to-end service fails.
Governance should include a service integration function or equivalent that owns the end-to-end service and the multi-supplier incident process. That function should maintain a shared severity matrix, contact and escalation lists, a common data format for incident telemetry, and regular joint exercises. The bank should test a multi-supplier major incident before the next month-end, including the regulator notification path.
The aim is not to assign blame but to make cooperation the rational choice. If suppliers are contractually required to share evidence and join a single bridge, the finger-pointing seen here becomes less likely. The post-incident review should produce owned actions across the bank and its suppliers, with the service owner accountable for the end-to-end improvement and the problem management practice tracking systemic causes.