Resolving conflicting risk ratings on a claims intake replacement
Northbridge Mutual is replacing the intake module of its claims platform. A risk workshop with four business and three technical stakeholders produced a 5x5 matrix, but two entries are disputed. For "duplicate claim not detected", business rated impact 5 and likelihood 2 (risk 10); the technical team rated impact 3 and likelihood 4 (risk 12), arguing that the matching rules are complex and the interface to the policy system is new and immature. The second entry, "claim PDF rendering slow", scored 6 overall and sits in the backlog with no test effort assigned. Testing must finish in three weeks. The full regression suite is 850 cases and the team estimates five weeks to run all of them. Two of the six testers joined last month. Historically, 60% of field defects traced back to the duplicate-matching and payment-calculation modules. The test manager proposes averaging the two duplicate-claim scores to 11, and dropping 400 regression cases that have passed unchanged for two releases.
1. How should the team resolve the conflicting ratings for "duplicate claim not detected", and how should the resulting risk level drive test effort?
Read the worked answer
Averaging the two scores is the wrong move: it hides that business and technical stakeholders are answering different questions. Impact is a business judgement (financial loss, regulatory breach, reputational damage); likelihood is a technical judgement (defect-proneness of the matching rules, immaturity of the new policy interface). The risk level is the product of both, so the two groups should not each produce a full score.
Resolve it this way:
- Fix impact from the business side only, using documented impact categories, and record the rationale.
- Fix likelihood from the technical side only, using evidence: complexity of the matching rules, defect history of the module, novelty of the integration.
- Where the groups still disagree, escalate to the named risk owner rather than splitting the difference; record the decision and the assumption.
Once settled (plausibly impact 5, likelihood 4 = 20, i.e. the highest band), the risk drives intensity, not just case count: more detailed test design (decision tables for the matching rules, boundary values on identifier fields), independent review of the matching specification, end-to-end tests across the new policy interface, and exploratory sessions in that area. The low-scoring PDF rendering risk can legitimately receive thin coverage, but it should be re-scored if performance becomes contractual.
2. Which risk mitigation activities, other than running more test cases, should the team apply within the three-week window, and why are they appropriate here?
Read the worked answer
Risk mitigation is not the same as more dynamic testing; some risks are cheaper to reduce by other means, and some residual risks must be managed rather than tested away.
Appropriate actions:
- Static techniques on the duplicate-matching module: code review and static analysis of the matching rules, because the defect history and complexity point to defect-prone logic that reviews find earlier and cheaper than system tests.
- Interface contract testing with the policy system team: the novelty of the interface is the main driver of likelihood, and contract tests catch schema and semantic mismatches before end-to-end runs.
- Monitoring and detection in production: logging and alerting on duplicate-claim overrides and manual corrections, so the residual risk of undetected duplicates is detected quickly rather than silently paying out twice.
- Phased rollout with a pilot branch: limits the impact if the matching logic fails in a way testing did not predict.
- A documented manual workaround and runbook for claims staff, so impact is reduced even if the defect escapes.
These are justified because the risk is high-impact and the three-week window caps how much dynamic testing can be done; the team should also be explicit about residual risk accepted at sign-off, since two of six testers are inexperienced and will not fully cover the matching logic alone.
3. Evaluate the proposal to drop 400 of the 850 regression cases. What should determine which cases are removed?
Read the worked answer
Dropping 400 cases may be defensible, but the stated criterion - "passed unchanged for two releases" - is not a valid one. A test that keeps passing is evidence of stability, not of low value; the modules with the worst field-defect history (duplicate matching, payment calculation) probably contain many of those long-green tests.
Selection should be risk-based and change-based:
- Change impact analysis: which code, interfaces and rules the intake replacement actually touches; tests covering untouched, stable areas are the safer candidates for removal.
- Traceability to risk: keep every case that covers a high or very high risk item, regardless of how long it has passed.
- Defect history: keep cases that have found defects in the last two or three releases, or that cover the modules named in the field-defect analysis.
- Coverage of the changed code: use structural coverage of the modified modules to check what would be left untested.
- Replace, don't just delete: move stable, low-risk cases into a less frequent regression cycle rather than deleting them, and keep a small smoke set that always runs.
With a 3-week budget against a 5-week suite, the team should also consider prioritising execution order so that high-risk cases run first if time runs short, and record the residual risk of the reduced suite for the release decision.