Defect Clustering in Release 7.3 - Acting on the Numbers
At a 40-person insurance software company, the test manager reports on release 7.3: 214 defects were found in system testing, 148 of them in the two premium-calculation modules, which make up roughly 15% of the codebase. The other twelve modules produced 66 defects between them. The release manager proposes: 'Next time, test only the two premium modules properly and smoke-test the rest - that is where the bugs are.' Separately, the CTO asks why the team should keep writing new tests for the policy module after two months of runs with zero failures, saying 'no defects found means the module is good.' The team has never documented a stopping rule and treats 'all planned test cases executed' as the exit criterion.
1. Evaluate the release manager's proposal. Which testing principle is being misapplied, and what would you recommend instead?
Read the worked answer
The relevant principle is defect clustering: a small number of modules usually contain most of the defects, and defect density data should steer effort. The release manager is applying it correctly up to a point - deeper testing (and more static analysis and review) in the premium modules is justified - but then overreaches. Clustering is a heuristic about where to look harder, not evidence that the other modules are safe.
What the data does not tell you:
- Why the difference: complexity, change volume, developer experience, or a shared component used by all modules.
- Whether a 'quiet' module was actually exercised by meaningful tests.
- Whether a recent change in premium calculation altered data contracts used elsewhere.
Recommendation: re-assess product risk per module using change size, technical complexity, usage frequency, business/financial impact and historical defect density - not defect count alone. Keep proportionate depth in premium modules, add depth wherever interfaces changed, and keep a risk-based regression sample elsewhere. A smoke test of a payouts module is not proportionate to its impact.
2. How should the CTO's statement be answered? Name the principles involved and what you would actually do about the policy module.
Read the worked answer
Two principles apply.
- Testing shows the presence of defects, not their absence. A test can only reveal the defects it is designed and able to detect. Two months of zero failures is at least as likely to mean the tests are no longer finding anything as it is to mean the module is clean.
- Tests wear out (the pesticide paradox). Re-running identical test cases on unchanged data stops yielding new information. If the module's code, data or usage has changed and the tests have not, the zero is uninformative.
- Absence-of-defects fallacy is the related trap: a defect-free module can still fail to meet user needs.
Practical response before answering: confirm the tests actually run and are not skipped or blocked; check what coverage the existing cases give over the module's decisions and interfaces; check whether the module changed in the last two releases. Then act: add new data and scenarios, review the cases against the risk profile, and refresh them. The honest answer to the CTO is that zero failures measures the tests, not the module.
3. The exit criterion is 'all planned test cases executed'. Critique it and propose a better stopping rule for this team.
Read the worked answer
It is an effort measure, not a quality or risk measure. It is satisfied by executing 100% of a weak, outdated set of cases, and it says nothing about whether the highest-risk behaviour was covered, whether open defects are acceptable, or what risk remains. It is also easy to game: execute and 'pass' the easy cases, and the criterion is met.
A better rule, agreed at the start of the release and refined as risk changes:
- Coverage of risk, not counts: all test conditions for the high-risk modules (premium calculation, interfaces that changed) executed and passed at the agreed technique coverage level (e.g. boundary values, decision branches, state transitions).
- Defect-based: no open critical or high-severity defects; defect detection rate trending down over the last two cycles; all fixed defects confirmation-tested plus regression around them.
- Residual risk documented: a written list of what was not tested and why, accepted by the business owner.
- Evidence of test quality: tests reviewed, not just run.
Also record when the criterion is not met and who accepted the deviation - otherwise 'all executed' silently becomes the answer again.