Monday Morning Payment Latency Recurrence
MetroPay processes card authorizations for 1,400 merchants. Every Monday between 08:55 and 09:20, authorization latency rises from 300 ms to over 8 seconds and 12–15% of requests time out. The service desk logs 30–40 tickets each week, mostly from merchants. Operations restores service by restarting application nodes; the incident is closed each time. A problem ticket has been open for six weeks with no root cause. Monitoring alerts only on CPU, not latency, so the major incident process is triggered by merchant complaints. A recent change moved the settlement batch job from 02:00 to 08:50 and introduced a new autoscaling policy. The problem manager wants to declare a major incident every Monday; the service owner says that would be noise because the workaround is known.
1. Is this best managed as a major incident, a problem, or both? Explain how you would prioritize.
Read the worked answer
It requires both, but they answer different questions. Incident management restores service and communicates with affected merchants; problem management finds and removes the recurring cause. The fact that a workaround exists does not make the event less of a major incident if impact thresholds are met: 12–15% timeout rates across 1,400 merchants is significant customer and revenue impact. Major incident declaration should be driven by impact criteria such as failed authorization volume, merchant count, and duration, not by whether the service desk has received complaints.
However, declaring a fresh major incident every Monday without changing the response would be noise. The better model is a pre-declared recurring major incident with a standing bridge, pre-agreed communications, and a known-error workaround activated before 08:55. The problem record should be prioritised high because the event is predictable, recurring, and financially material.
The service owner is right that the workaround is known, but wrong to treat that as closure. Known error does not mean known cause.
2. What immediate and longer-term actions would you take, including known error and workaround management?
Read the worked answer
Immediately, create a known-error record with a documented workaround: pre-scale application nodes before 08:45, shift or throttle the settlement batch job, and monitor latency directly. Add latency and timeout thresholds to monitoring so the team is alerted before merchants complain. Give the service desk a script and merchant communication template. During the Monday window, staff a named incident owner and review the bridge proactively.
Longer term, investigate the interaction between the 08:50 batch job and the new autoscaling policy. The timing correlation is strong: moving a resource-intensive batch into peak authorization hours can create CPU and connection-pool contention, and autoscaling may react too slowly or restart nodes rather than add capacity. Test the batch job in isolation, compare node and database metrics, and reproduce in a pre-production environment.
Do not close the problem record merely because the workaround succeeds. Close it when the root cause is removed or the risk is formally accepted with monitoring. Link the problem to the change that moved the batch job and to the autoscaling change.
3. What data and change links would you require before closing the problem record?
Read the worked answer
Require evidence that the failure mode is understood, not just suppressed. That means time-series data for latency, timeout rate, CPU, memory, database connections, and batch duration across several Mondays, compared with a control day. Include node restart times, autoscaling events, and merchant impact counts. Show whether the workaround reduced impact and whether any residual risk remains.
Link the problem record to the settlement batch change and the autoscaling change, including approval dates and configuration differences. Confirm whether the issue existed before those changes. If the root cause is confirmed, record the fix, the verification test, and the monitoring change that would detect recurrence. If it is not confirmed, record the accepted risk, the owner, and a review date.
Finally, check that incident and problem records are linked so trend analysis is possible. Closure should require service owner sign-off and evidence that the known error is either eliminated or controlled.