The 02:15 payroll bridge and unclear decision rights
At 02:15 a payroll provider's payment run fails for 200 client organizations, affecting 48,000 employees. The service desk analyst on shift is the designated incident manager, but she has never run a major incident. The technical lead starts directing six engineers in a chat channel and ignores the incident manager's request for a status update. The service owner for payroll joins, takes control of the engineers, and begins approving database changes herself. An outsourced infrastructure supplier is responsible for the failing database cluster but will only engage through its contracted service desk, which opens at 03:00. Client-facing communications have not started. The contract has separate SLAs for the provider and the supplier, but no end-to-end restoration target. The incident manager wants to escalate to the on-call major incident manager, but the technical lead says escalation will 'slow us down'. Restoring service is urgent: payroll must be processed before 06:00.
1. Analyse the role confusion in this major incident. Who should hold decision rights for coordination, technical recovery, and business communication, and what should the incident manager do about the technical lead and service owner?
Read the worked answer
The incident manager is accountable for coordination: declaring and running the major incident, maintaining the timeline and decision log, driving escalation, and ensuring communications. She is not accountable for technical diagnosis. The technical lead should own technical recovery, including the recovery plan, engineer allocation, and technical status updates. The service owner should own business alignment and service outcomes, including client impact decisions, but should not bypass technical authority or change control by approving database changes directly.
The junior analyst should escalate to the on-call major incident manager immediately. The technical lead's objection does not override a pre-agreed escalation policy; a major incident affecting 48,000 employees is exactly when additional coordination capacity is needed. The on-call manager can chair the bridge while the original incident manager keeps the timeline.
The service owner's database approvals are a control risk. Any change should go through the emergency change route with a named technical authority and documented risk, even if that takes minutes. The incident manager should state the decision rights out loud on the bridge, ask each role to confirm, and record it. If the technical lead continues to bypass, the on-call manager escalates to the technical lead's line manager. Role clarity is not bureaucracy here; it prevents duplicated work and uncontrolled changes.
2. Describe the value stream you would follow from 02:15 to service restoration, including supplier engagement and escalation. Identify where the current flow breaks.
Read the worked answer
The value stream is: detect and confirm the failure; log and classify; prioritise using impact and urgency; declare a major incident; assemble the response; diagnose and restore; validate recovery; communicate; close; and learn. In this case, detection has happened through the failed payment run, but logging and classification are informal, and there is no single major incident record.
The flow breaks at several points. The incident manager lacks the authority and experience to run the bridge. The technical lead works in a separate chat channel, so the timeline and decisions are fragmented. The service owner bypasses change control. Supplier engagement waits for a contracted service desk to open, even though the major incident should trigger an out-of-hours escalation clause. Communications are not started, so clients learn nothing.
I would create one bridge and one record, with the on-call major incident manager chairing. The technical lead runs a technical swarm and reports status at fixed intervals. The supplier manager invokes the contractual duty manager immediately and does not wait until 03:00. A communications lead drafts a holding statement for clients and internal stakeholders. The service owner confirms business priority and any client-specific constraints. A scribe records decisions and timestamps. Restoration is validated against the payment run, not just database availability. The incident closes only when payroll processing is confirmed, with a post-incident review on role clarity and supplier escalation.