A duplicated settlement batch at a 40-person payment institution
PayNow Europe is a licensed payment institution processing 1.4 million card and account-to-account transactions daily for 900 merchants. On a Friday at 19:42 a database failover in its primary data centre triggered duplicate settlement files. Between 19:42 and 22:20, 186,000 transactions were either declined or settled twice, affecting 61,000 end users. The incident was detected at 21:05 when the merchant support inbox received complaints; monitoring dashboards showed green because the synthetic transaction test covered a different payment scheme. The on-call engineer logged it as a high-severity internal incident at 22:40, after service had been restored at 22:20 by rolling back the settlement batch. The same failure occurred twice earlier in the quarter with smaller impact. The institution has 40 staff, outsources its data centre to a provider in another Member State, and reports to its competent authority. Its incident policy was written in 2021 and defines major only by duration exceeding four hours.
1. Should this be classified as a major incident? Walk through the criteria and identify the process failures separately from the classification decision.
Read the worked answer
Likely yes, and the reasoning must be documented against the criteria rather than the internal four-hour rule, which is inconsistent with DORA because duration is one criterion among several.
Applying the classification criteria:
- Clients affected: 61,000 end users and 900 merchants, 186,000 transactions. Significant both absolutely and relative to the institution's book.
- Duration and downtime: 2h38m of degraded settlement on a payment service, during which transactions were declined or wrongly settled.
- Data integrity: duplicate settlement files mean incorrect processing, not mere unavailability. Integrity events are usually weighted more heavily than availability events because of knock-on credit, liquidity and dispute exposure.
- Economic impact: double settlements create real financial exposure; quantify the value at risk, do not leave it qualitative.
- Criticality of the service: payments are core to the licence.
- Reputational: detection through complaints worsens the assessment.
- Recurrence: third occurrence in a quarter is an aggravating factor and belongs in the classification record.
Separate process failures:
- Detection came from the merchant support inbox, not monitoring. The synthetic test covering only one scheme is a detection design flaw under Art. 10 and should be treated as a control gap, not bad luck.
- Classification took 95 minutes after restoration, indicating the on-call engineer had no decision support and no authority clarity.
Action: classify as major, notify, and rewrite the policy so duration is replaced by a multi-criteria decision tree with a lower threshold for integrity events and repeat occurrences.
2. Set out the reporting timeline and content, including what must be preserved as evidence and how the outsourced data centre provider fits in.
Read the worked answer
Work from two clocks and preserve the evidence of when each started.
- Detection/awareness: 21:05 from the first complaint. Record the moment of awareness and its basis; supervisors test whether detection was unreasonably late, and here it plainly was relative to the failure start at 19:42.
- Classification: without undue delay. If classified major at, say, 23:00, the initial notification is due within four hours of that classification and in any event no later than 24 hours from detection. The four-hour window is the target; the 24-hour limit is an outer bound.
- Initial notification: entity identification, incident type and classification, detection and classification times, affected services and clients, the assessed impact, and whether the incident is ongoing. Do not wait for root cause.
- Intermediate report: within 72 hours of the initial notification, updating impact, root cause where known, and recovery measures taken.
- Final report: within one month of the intermediate report, covering root cause, actual impact, corrective actions and lessons learned.
Provider dimension: request the data centre provider's incident information immediately and contractually; name it in the report. If its failover procedure caused the duplication, its service levels, notification obligations and periodic review must reflect that.
Evidence set: the classification record, the named decision-maker, timestamps, copies of every submission, and the internal decision log. If the classification is later revised downward, document the reasoning rather than quietly retracting the notification. For a 40-person institution, template the report fields in advance so the on-call engineer is filling blanks at 23:00, not writing prose.
3. The failure has now occurred three times in one quarter. What changes in your response, and how does it affect the provider relationship?
Read the worked answer
Three occurrences turn an incident into a systemic defect. The response shifts from restore-and-close to design-and-verify.
- Root cause at design level. Duplicate settlement files after failover suggest recovery was designed for availability, not for idempotency. Fix structurally: idempotent batch keys, pre-settlement validation, reconciliation controls before files are released.
- Detection. Extend synthetic monitoring to all schemes and add a duplicate-batch-identifier alert. Customer complaints must not be the primary sensor.
- Corrective actions from the first two events. If none were raised, or raised and not closed, that is the real finding. Assign owners with budget authority, set dates, define an escalation rule by age, and report repeat incidents to the management body.
- Learning and evolving (Art. 13). Post-incident reviews with the provider present; feed the outcome back into the risk assessment, the control set and the testing plan.
- Provider relationship. If the failover runs on the provider's infrastructure, its incident notification and root-cause obligations must be contractual and tested. Its performance feeds the periodic review, the concentration assessment and, if severe, exit planning. Escalate to its risk function, not only to account management.
- Regulatory posture. Disclose the earlier occurrences in the final report rather than letting the authority discover a pattern; voluntary transparency on recurrence is treated far better than apparent under-reporting.
- Size reality. Forty staff means the fix must be automated. A manual checklist will fail on the fourth occurrence.