Recurring Payment Gateway Timeouts Without a Known Error
Over five weeks the CI 'PGW-PROD-02' has produced 22 incidents with the same symptom: gateway timeout during peak load. Average time to restore is 40 minutes, and every one of them is fixed the same way - restarting the service. Three of those incidents breached the 4-hour resolution SLA. Two agents know they can route traffic to PGW-PROD-03 while the restart runs, but that workaround lives only in their personal notes. No problem record exists and there is no knowledge article. The service owner wants to keep treating these as incidents because 'they always get fixed.' The problem manager wants to close the loop quickly so the ticket count stops growing.
1. What evidence justifies opening a problem, and what should the problem record be linked to?
Read the worked answer
Open the problem on the strength of four signals, not on a single bad incident:
- Recurrence - 22 incidents in five weeks on one CI with one symptom is a trend, not noise.
- Same workaround every time - restart is a mitigation, so the underlying fault is never removed.
- SLA breaches - three breaches is measurable business impact.
- Undocumented knowledge - the workaround exists only in two agents' heads, which is itself a risk.
Link the problem to the CI (PGW-PROD-02), to the 22 incidents as related records, and to affected CIs including PGW-PROD-03. Keep the incidents intact as the operational record - the problem is an additional record, not a conversion. Assign problem tasks so root cause analysis and the workaround write-up can proceed in parallel.
2. At what point can this become a known error, and what does that unlock?
Read the worked answer
A problem becomes a known error once the root cause has been identified and a workaround has been validated - the permanent fix does not need to exist yet. That is the whole point of the distinction: known error means 'we understand it and we can tell people how to survive it.'
The sequence here is:
- Problem moves New -> Assess -> Root Cause Analysis.
- During RCA the two agents' restart-and-failover procedure is written down and tested.
- Root cause is confirmed (for example, a connection pool leak under peak load).
- Mark the problem as a known error and publish a knowledge article containing the symptom, the workaround steps, and the affected CI.
What that unlocks: incident agents get the article suggested on matching incidents, mean time to restore drops, and the 40-minute restarts stop being tribal knowledge. The problem record stays open and moves to Fix in Progress.
3. The service owner proposes closing the problem once the workaround article is published. Evaluate that proposal.
Read the worked answer
Reject it. Publishing a workaround satisfies the known-error criterion; it does not satisfy closure.
A problem is closed when the root cause has been removed and the fix verified, or when a deliberate, documented decision is taken that the risk is acceptable. A restart workaround leaves the fault in place: incidents keep arriving, SLAs keep breaching, and the ticket volume the service owner wants to reduce will not fall.
What to do instead:
- Keep the problem in Fix in Progress with a named owner and target date for the permanent fix.
- Publish the workaround article now - that is a win, just not a closure.
- If the fix genuinely cannot be funded, record the decision and a review date on the problem rather than closing it, so the risk is visible instead of invisible. Closing on a workaround is how an organisation ends up with 22 incidents that nobody is accountable for.