A New Molecular Assay and the Validation Question
A laboratory plans to implement a laboratory-developed multiplex PCR panel for a set of respiratory pathogens. The laboratory has 40 residual clinical specimens, of which 12 are positive for at least one target by a reference method. The director proposes to validate using those 40 specimens, compare against the reference method, and report accuracy. A technologist notes that several targets have only 1-2 positive specimens, and that the laboratory has no proficiency testing material for two of the targets. The laboratory also wants to report a limit of detection, which has not yet been established.
1. Critique the proposed validation design and identify the gaps.
Read the worked answer
The design has several gaps. First, 40 specimens with only 12 positives is a small sample, and for several targets the positive count (1-2) is far too low to estimate sensitivity with any meaningful confidence interval-a single discordant result would swing the estimate wildly. Second, comparison against a reference method is only as good as the reference method; discordant results need resolution (e.g., a third method or sequencing), not automatic scoring against the comparator. Third, no limit of detection has been established, yet the laboratory wants to report one-LOD must be determined experimentally (e.g., limiting dilution of quantified material) and is a separate validation element. Fourth, there is no proficiency testing for two targets, which must be addressed with an alternative assessment (e.g., blind samples, interlaboratory comparison, or known-characterized material).
The director's proposal conflates 'we ran some specimens' with validation.
2. Propose a defensible validation plan.
Read the worked answer
A defensible plan includes: (1) establish the LOD for each target by limiting-dilution testing of quantified material, with replicates; (2) supplement the clinical specimens with characterized material (inactivated cultured organisms or synthetic targets) to reach an adequate number of positives per target-typically at least 20-50 positives per analyte is a reasonable target, with the exact number justified; (3) include negatives that challenge specificity, including near-neighbors and specimens with other pathogens; (4) resolve discordant results with an independent method; (5) address the two targets lacking proficiency testing through an alternative assessment and document it; (6) define acceptance criteria before testing and report accuracy with confidence intervals; (7) document the validation in a report reviewed and signed by the director.
The guiding principle is that the validation must be proportionate to the claims being made. A panel that reports multiple targets needs evidence for each target, not just an aggregate accuracy figure.