SAL1 study materials
SAL1 study resources

Three simultaneous escalations, six analysts

SAL1 · Put your knowledge to work in context.

All case studies

Three simultaneous escalations, six analysts

You are duty manager for a 24/7 service desk with six analysts on shift. At 09:12 a P1 is raised: the payment gateway is returning errors on roughly 30% of transactions. At 09:19 a second alert fires: the payroll batch that produces Friday's payslips has failed; today is Wednesday and 2,400 staff are affected. At 09:25 the CIO's assistant calls: the CIO cannot log in to email and has a board meeting at 10:00. Your two strongest analysts are already committed to the gateway. The gateway vendor's support line quotes a 45-minute callback; the payroll vendor answers immediately. Neither incident has a runbook. The next scheduled update to the business is 11:00.

1. How do you sequence and staff these three, and why?

Read the worked answer

Rank on impact, reversibility and clock - not on who shouted

  • Payment gateway stays the top incident. Revenue loss and customer-visible failure, ongoing, and it already has your two most capable people. Do not pull them.
  • Payroll is second but has a hard time trigger. Thirty percent of transactions failing is a per-minute cost; a missed Friday pay cycle is a per-cycle failure that cannot be recovered inside the same period. Give it the third-strongest analyst, set an explicit trigger - if not resolved by early afternoon, promote it and add a senior - and use the fact that the payroll vendor answers immediately. Note the sequencing insight: the gateway vendor's 45-minute callback is not a reason to sit idle, and the payroll vendor's availability is not a reason to staff payroll heavily. Vendor responsiveness does not set your priority.
  • CIO email is not an incident. Verify identity and handle it at the service desk. Being the CIO changes the communication, not the queue position. Leaving the assistant without an answer is the actual failure mode - give a realistic ETA and, if useful, an alternative access route for the board pack.

Keep one analyst free as a floater. With both incidents runbook-less, the riskiest state is everyone committed and nobody available to be pulled to a third failure.

2. It is 09:30. What do you communicate, to whom, and what do you deliberately leave out?

Read the worked answer

Three audiences, three messages, one set of facts.

  • To the floor: who owns each incident, what is known, what each owner is doing next, and the next internal checkpoint (say 10:15, not 11:00 - the external update cadence is not the internal one).
  • To the payroll manager and finance: confirmed facts only, the impact window, and when you will next speak. No cause, no ETA you cannot hold. Someone in finance needs to start thinking about a manual contingency now, and that only happens if you tell them early.
  • To the CIO's assistant: one line - identity check needed, likely cause, ETA, alternative access.

What you leave out: unconfirmed root cause, any number you have not verified, and the word "resolved" until it is. At 09:30 you almost certainly do not know why the gateway is failing on 30% of transactions, and a wrong cause stated once costs more time than it saves.

Also decide who is not being updated yet - the wider business - and keep to the 11:00 commitment with them so the cadence means something.

3. The gateway incident turns out to be a configuration change deployed at 08:55 by another team who did not raise a change. What do you do differently for the rest of the day?

Read the worked answer

The technical path is now short - roll back if the change is reversible and the rollback is safer than the forward fix. The judgement is in what you do with the rest of the day.

  • Confirm the rollback actually restores service on a sample before declaring anything; a 30% error rate has a tail, and early metrics will flatter you.
  • Push payroll hard now, because you have just freed senior capacity and the payroll clock has not moved.
  • Record the unauthorised change contemporaneously - who, when, what approval was missing - while people remember. Reconstructing it next week produces a report nobody trusts.
  • Do not run the change-management argument today. Blame allocation during an active incident suppresses the disclosure you need for the next one. Handle it as a separate conversation with that team's manager once both incidents are closed.
  • Ask the question that matters for tomorrow: what detection would have caught an unapproved change before customers did? That is the finding, not the individual.