Defending a Sample When 78% of Complaints Are Closed Without Investigation
MediFlow manufactures infusion pumps on three product lines. You are lead auditor for a 4-day on-site audit of complaint handling and CAPA, accompanied by one trainee. Volumes: 11,400 complaints in the last 12 months; 62 open CAPAs, the oldest open 410 days; three CAPAs that were closed and later reopened. The complaint log shows 78% of complaints closed within 5 days, most coded X9 ("no investigation required"); 41 of the last 300 X9-coded complaints contain the phrase "device stopped infusing." Two MDRs were filed last year; one was submitted 9 days after the firm's own 30-day deadline. The previous audit raised one minor nonconformity in complaint trending, closed as corrected. The quality director says, "Sample whatever you like, we have nothing to hide," and asks you not to "get lost in old files." Your audit plan allows 22 hours of interview and evidence time.
1. Design a defensible sampling plan. Specify the strata, the approximate sample sizes, and why each stratum is included.
Read the worked answer
Principle: audit sampling is risk-based and purposive, not statistical acceptance sampling. The objective is to test whether the system works, so the sample must over-weight where the system is most likely to fail. Document the population, criteria, method and size before selecting.
Strata and rationale:
| Stratum | Approx. n | Why |
|---|---|---|
| X9-coded complaints containing "device stopped infusing" | 20–25 | Highest risk of an under-investigated complaint that is actually reportable; the triage rule is the suspected systemic weakness |
| X9-coded, random within the remaining X9 population | 20–25 | Tests whether the pattern is real or an artifact of keyword selection - protects against cherry-picking |
| Complaints involving injury, death, or malfunction with patient harm | 10–15 (or all in a defined window) | Consequence severity; tolerates no sampling error |
| Complaints closed near the MDR decision deadline | 10 | Tests timeliness of the regulatory decision, not just the filing |
| The late MDR plus both filed MDRs | 3 (all) | Known failure; examine causation |
| Open CAPAs >90 days and all reopened CAPAs | 10–12, including 3 reopened (all) | Effectiveness and ageing |
| Random closed CAPAs | 8–10 | Representativeness of the CAPA population |
Then follow trails: each sampled complaint → investigation record → risk file reference → CAPA → effectiveness check. Sampling is a starting point; a single trace can generate several findings.
Do not: use a purely random sample of 20 closed complaints as the trainee suggests - random sampling of a population that is 78% filtered by an untested triage rule mostly re-tests the easy 78%.
2. Your trainee proposes pulling 20 random closed complaints "to be objective" and wants to interview the complaint manager alone. How do you handle this, and what does it reveal about auditor competence?
Read the worked answer
Correct the reasoning, not just the decision.
- "Objective" does not mean "random." Random selection is one tool for representativeness; it is the wrong tool when the audit question is "does the X9 triage rule work?" A random sample would under-cover the highest-risk sub-population precisely because it is small relative to the total. Explain the distinction between representativeness (random) and risk coverage (purposive), and let the trainee select the random within-X9 sample themselves so the reasoning is practiced.
- Interview conduct: the trainee does not interview the complaint manager alone, not because trainees are untrustworthy, but because (a) the trainee has not yet demonstrated the competence to probe a defensive answer, and (b) the lead auditor needs first-hand evidence to classify findings. The trainee observes, takes notes, and drafts questions reviewed in advance.
- Competence point: ISO 19011:2018 treats auditor competence as demonstrable knowledge and skill, including sampling judgement and interviewing technique. Supervised practice with feedback is how competence is built; unsupervised substitution is not. Record the trainee's role in the audit plan.
- Also flag: the quality director's "don't get lost in old files" is an attempt to scope-limit the audit. Politely restate that the audit scope includes complaint records for the defined period and that selection is based on risk, then proceed. It belongs in the audit report as a note on access/cooperation only if it actually obstructed evidence gathering.
3. Your plan yields only the samples you actually review. How do you record sampling so that your conclusions are proportionate, and what can you legitimately conclude?
Read the worked answer
Document in the audit plan and report: population definition and period; strata and counts selected; selection method (random / purposive / 100% of a defined subset); basis for the size (risk, consequence, prior history); who selected; and the explicit limitation that findings relate to the items examined.
Proportionate conclusions - the reasoning chain:
- A defect in one sampled item, if it is an isolated execution lapse with no pattern, supports a minor nonconformity against the affected requirement.
- A defect that traces to the rule (e.g., the X9 triage criterion itself permits closure without investigation) is a systemic finding regardless of sample size - you are not extrapolating from a count; you are evaluating the criterion against the requirement. This is the key distinction auditors often miss: system-level findings rest on the design of the control, not on statistical incidence.
- A defect found in 4 of 20 X9 complaints, plus a documented criterion that permits it, supports a statement about the criterion and about the risk of the unsampled population - but not a precise rate, and never a statement such as "approximately 20% of complaints are mishandled."
- Zero defects in 20 items does not prove the population is conforming; state the sample as a limitation.
Report language: "Records selected and examined: n = 88 across six strata (see appendix). Nonconformity is based on the sampled records identified in the finding statement and on the documented triage criterion, not on statistical projection." This survives notified body or FDA scrutiny; an unqualified percentage claim does not.