I85PREP
⌘ K
QAI CAST study materials
QAI CAST study resources

QAI CAST: Test Measurement and Metrics

QAI CAST · Work through your exam, one topic at a time.

All guide sections

Test Measurement and Metrics

Test Measurement and Metrics asks whether testing is effective and whether you can prove it. CAST tests the meaning and arithmetic of common measures, plus the judgement to use them without distorting behaviour. You should be able to calculate defect density and defect removal efficiency and explain what each number does and does not tell you.

Core measures

  • Test execution progress: cases executed versus planned, and passed versus executed.
  • Defect density: defects per KLOC, per function point, or per module. Always state the denominator.
  • Defect removal efficiency (DRE): defects found in a phase divided by (defects found in that phase plus defects found later), expressed as a percentage.
  • Defect leakage or escape rate: defects found in production divided by total defects found.
  • Coverage: requirements covered by at least one test case; code coverage at statement, branch, or path level.
  • Reliability: mean time between failures (MTBF) and availability.

Using metrics well

Anchor each metric to a goal - the Goal–Question–Metric approach. Trends over several iterations are more informative than a single reading. Never make a metric a personal target: defect-count targets encourage under-reporting, and DRE targets encourage gaming the phase boundaries.

Worked example

A team finds 80 defects during system test. Customers report 20 defects in the first six months after release. What is system-test DRE, and what is the leakage rate?

Reasoning: DRE = 80 / (80 + 20) = 80%. The 20 escaped defects represent a leakage rate of 20 / 100 = 20%. Now suppose a design review is added earlier and finds 30 defects before system test. System-test DRE stays 80% for that phase, because those 30 defects were removed earlier and are not part of the system-test filter calculation. If you simply added 30 to the numerator you would overstate the phase's effectiveness.

Traps that cost marks

  • Reporting defect density without stating the size unit or module scope.
  • Treating a rising defect count as automatically bad; it may reflect better detection.
  • Comparing DRE across projects of very different complexity or domain.
  • Including invalid, duplicate, or rejected defects in counts.
  • Using a single snapshot instead of a trend to judge quality.