Eight Decision-Table Tests and Two Production Defects
Northbridge Insurance is testing a premium calculator. The rating rules use two conditions - driver age (bands: under 25, 25 to 64, 65 and over) and claims in the last three years (0, 1, or 2 or more) - plus a loyalty discount applied when the policy has been held for 5 years or more. The tester derived eight test cases from a decision table, ran them all successfully, and reported 96% statement coverage and 78% branch coverage from the coverage tool; the remaining 4% of statements sat in an exception handler the tester considered unreachable. On release, two production defects appeared: a 25-year-old with one claim was charged the under-25 rate (the code reads 'if age > 25'), and a policy with a blank claims-history field crashed the batch run when the field was null rather than zero. The test lead's summary stated that the rating module was 'fully covered by decision-table testing'. The release is now in a two-week hardening window and the product owner wants to know whether the module can be considered sufficiently tested.
1. Explain, using the two production defects specifically, why the decision table and 96% statement coverage did not provide the assurance that was claimed.
Read the worked answer
The decision table tests rules, not values. Eight rules means eight representative input combinations. If the 'age 25 to 64' rule was exercised with a 40-year-old, the test passes whether the code says 'age > 25' or 'age >= 25'. The table cannot see the difference because it deliberately collapses a range into one representative. The boundary between 'under 25' and '25 to 64' is exactly where the defect lives, and no decision-table test is obliged to land on it.
Statement coverage does not require branch outcomes per boundary. Ninety-six percent statement coverage says the statements ran, not that both outcomes of each decision were taken for each boundary value. The number that mattered was 78% branch coverage - 22% of branch outcomes were never exercised. An off-by-one comparison sits precisely in an untaken branch outcome.
The decision table models valid combinations only. A null claims-history field is not a rule in the table; it is an invalid input class. Decision-table testing by construction tests the specified, valid behaviour, so the null path was never entered - and the untested 4% 'unreachable' exception handler is the obvious suspect for the crash. 'Unreachable' was an assumption, not a measurement.
The claim itself conflated two things. 'Fully covered by decision-table testing' is true at the level of the rules and false at the level of the input domain. A technique is exhaustive only within the model it uses; the model here had no boundaries and no invalid classes.
2. Specify the additional test design work you would do. Name the techniques, give the number of tests you would add, and state what each group targets.
Read the worked answer
1. Boundary value analysis on every boundary, three values each.
- Age: 24, 25, 26 and 64, 65, 66.
- Claims: 0, 1, 2 with the overflow at the maximum permitted value.
- Loyalty: 4, 5, 6 years.
Hold the other conditions at a neutral value and vary one boundary at a time: roughly 10 tests. These directly target the '>' versus '>=' class of defect.
2. Boundary interactions where two conditions meet. Age boundary crossed with loyalty boundary (25 and 5 years, 24 and 5, 26 and 5): 4 tests. These catch errors that only appear when two boundary decisions are combined.
3. Invalid equivalence partitions. Null claims field, empty string, non-numeric age, age 0, negative age, claims count above the stored maximum, missing loyalty date: about 7 tests, each expecting a defined error response rather than a crash. This is the group that would have caught the batch failure.
4. Force the exception handler. One test that provokes the null path deliberately, to convert 'unreachable' from an assumption into evidence.
5. Coverage targets. Re-run with 100% branch coverage excluding branches proven unreachable by inspection, and use condition or MC/DC coverage on the age and claims decisions if the tool supports it - the defect was a single condition error inside a compound decision, which branch coverage alone can miss.
Total: roughly 22 additional tests, concentrated where the two defects were, not spread evenly. I would also add both production defects as permanent regression tests and scan the rating code for other comparison operators of the same shape, since one off-by-one usually has siblings.
3. Recommend how to state residual risk to the product owner and what coverage evidence to put in front of them.
Read the worked answer
Do not repeat 'fully covered'. The honest statement has three parts: what was tested, what was not, and what that means in business terms.
Residual risk statement. 'Valid-path rating logic is now covered by decision-table rules plus boundary and invalid-input tests, with branch coverage at X%. Two defect classes were found late because the original suite had no boundary or invalid-input tests. They are now covered by regression tests. Remaining unverified areas: the exception handler (provoked once, not exhausted), combinations of three or more conditions, and any rating rule not represented in the decision table.'
Evidence to present.
- The coverage report, with every uncovered branch listed and a reason against each: unreachable by inspection, low risk, or untested.
- The boundary test list with the exact values used, so the product owner can see 24/25/26 explicitly.
- The two production defects reproduced as failing-then-passing regression tests.
- A short traceability matrix: rating rule to test case, so gaps are visible rather than asserted.
Recommendation. The module can be considered adequately tested for the valid-path rules once the boundary and invalid-input tests pass; residual risk sits in the untested combinations and the exception handler. For the handler, offer the product owner a choice: accept the documented risk, or fund one day to test and harden it. Either way, record who accepted what - a coverage number is evidence, not a release decision.