Black Friday checkout degradation and the P1 queue
An online retailer is 40 minutes into its Black Friday sale. Checkout p99 latency has risen from 0.8 s to 4.2 s, and cart abandonment is 12% above baseline. The database connection pool is saturated. At the same time the service desk has received 600 'Where is my order?' queries and 250 password-reset requests. A warehouse application has raised a disk-capacity warning at 82%, though picking is still running. The service desk lead wants every contact logged as a P1 incident to 'get visibility'. The on-call major incident manager is unsure whether to declare a major incident for checkout only, and whether the password resets indicate an authentication failure. The problem manager notes that connection-pool saturation was seen twice last month and was fixed by a restart. The retailer's major incident criteria say 'revenue-affecting checkout degradation' and 'any suspected data breach'. The CISO has asked whether this is a cyber attack.
1. Classify each of the four streams - checkout latency, order-status queries, password resets, and the warehouse disk warning - as incident, service request, problem, or event. Justify each classification and decide whether to declare a major incident.
Read the worked answer
Checkout latency is an incident: it is an unplanned reduction in service quality, even though checkout is still partly available. The business impact is direct revenue loss and reputational damage, and urgency is high because the sale is live. It meets the stated major incident criterion of revenue-affecting checkout degradation, so I would declare a major incident, appoint a major incident manager, open a bridge, and start stakeholder communications.
Order-status queries are service requests - requests for information - not incidents. They should go to self-service or the order team, not the incident queue.
Password resets are normally service requests. However, 250 in a short window may indicate an authentication service failure. I would compare the volume with baseline and check authentication error rates before deciding. If the authentication service is degraded, that is a separate incident; if not, keep them as requests.
The warehouse disk warning is an event: a change of state that does not yet affect service. It becomes an incident if capacity threatens picking. Because it could recur, it may also warrant a problem record.
The key judgement is that classification should follow definitions and business impact, not contact volume. A major incident is declared for checkout, not for the 850 requests.
2. The service desk lead wants every contact logged as a P1 incident. Explain the operational and data-quality consequences, and describe how you would manage the queue and linkages.
Read the worked answer
Logging every contact as a P1 incident corrupts the priority data used for triage, reporting, and SLA management. It creates false demand, pushes genuine P1s down the queue, triggers unnecessary escalation, and trains responders to ignore P1 alerts. It also makes later trend analysis unreliable because incidents and service requests are mixed.
I would apply the impact/urgency matrix and separate the streams. The checkout issue gets one major incident record with a clear impact statement, affected services, and a decision log. If individual users need tracking, they can be linked as child incidents rather than creating hundreds of unrelated P1s. Order-status queries go into a service request queue with bulk handling, self-service deflection, and a standard response. Password resets stay in the request queue unless authentication monitoring confirms an incident.
The major incident manager should communicate the classification rule to the service desk lead and explain that visibility comes from a shared dashboard, not from mislabelling records. During the incident, the priority is restoration, so the problem manager should capture evidence for a problem record on connection-pool saturation but not start deep investigation that competes for the same engineers. After restoration, review classification accuracy and the request-deflection options.