No Ground Truth in the Reading Room

Technique 1: multiple imperfect oracles with adjudication

Use three readers with majority or adjudicated consensus on the 500-image set, and report the human-human agreement (kappa) alongside the model's score. The agreement rate is a ceiling on what any measurement can resolve: if the radiologists agree only 92% of the time, a model that scores 95% against one reader is not meaningfully better than the reader. Reporting the model against a single reader's labels without that caveat is the classic mistake.

Limit: adjudication is expensive and does not create truth, only a better consensus.

Technique 2: high-confidence subset

Use the 140 CT-confirmed images as a high-precision gold set.

Limit: CT is ordered because of clinical suspicion, so this subset is enriched for positives and for severe cases. It is good for detecting gross errors and catastrophic false negatives; it is a biased basis for estimating sensitivity on the general screening population.

Technique 3: metamorphic testing

Relations that must hold without knowing the true label:

• horizontal flip preserves 'pneumothorax present' (the laterality field changes, the finding does not);
• rotation of +/-10 degrees, brightness/contrast shifts within acquisition norms, JPEG recompression, cropping of black borders - output should be stable;
• adding a realistic synthetic pneumothorax marker to a confirmed-normal image should flip the output to positive. Any violation is a failure, no oracle required. This scales to thousands of images that have no ground truth at all.

Limit: metamorphic relations test invariance and robustness, not diagnostic correctness. A model can be perfectly invariant and consistently wrong.

Technique 4: back-to-back testing against the standard of care

Shadow mode on live images; every disagreement with the radiologist's read is adjudicated. This measures the thing you actually care about - marginal value over current practice - not agreement with a stored label.

Restate the requirement first - as written it is untestable.

'Sensitivity >= 0.95' is meaningless without specifying: against which reference standard, on whose images, at which threshold, with what confidence. Rewrite as: sensitivity >= 0.95 against adjudicated reference reads, on local images, at the threshold selected for local prevalence, with the lower bound of the 95% CI at or above 0.95.

External validation is mandatory.

The vendor's evaluation used its own hospital's scanners. Different scanners, different referral mix - the model has not been tested on your distribution. Build a local reference set stratified by scanner and referral reason, with pre-specified enrichment for positives. At 1.5% prevalence you would need thousands of unselected X-rays to accumulate enough positives; enrichment is the only practical route.

The operating point is a decision, not a property of the model.

0.88/0.90 is one point on an ROC curve. Ask the vendor for the curve, then choose the threshold that achieves 0.95 sensitivity on local data and read off the specificity. Then compute the operational consequence using local prevalence: at 1.5% prevalence and 0.80 specificity, out of 1,000 X-rays you would get about 15 true positives and about 197 false positives. That is an alert burden, and it is the number the clinical lead has not seen.

Report PPV, not just sensitivity and specificity. PPV depends on prevalence and therefore changes between hospitals. A model validated at 8% prevalence will look far worse at 1.5% even with identical sensitivity and specificity.

Pre-agreed decision rule:

• accept only if the lower CI bound on local sensitivity >= 0.95 and alert burden is within radiologist capacity;
• reject otherwise, or accept only with a human read of every negative, which changes the workflow;
• never accept on pooled or vendor-internal data.

Also check: calibration, and performance disaggregated by scanner, patient position, body habitus, and presence of chest drains or other pathology.

No. Code coverage answers a question nobody is asking here.

The model is a numeric artefact; 100% statement coverage of the inference wrapper tells you the wrapper executes. It tells you nothing about whether the model has been exercised on the inputs that matter. The relevant coverage is input-space coverage, and the relevant question is which regions of the clinical input space have never been tested.

What to measure instead

A coverage matrix over the factors that plausibly change behaviour:

• acquisition: scanner model/vendor, portable vs stationary, exposure, patient rotation;
• patient: age band, sex, body habitus, presence of chest drains, tubes, or prior surgery;
• pathology context: pneumothorax alone, with effusion, with consolidation, post-trauma, post-procedure;
• image quality: artefacts, clipped lung fields, incomplete inspiration.

Report which cells have zero or very few test cases and treat the empties as test-design findings. Cells that are rare in the reference set but common in live traffic are the highest-risk untested regions.

Combine with metamorphic coverage - which transformations have been applied to which images - so that invariance has been probed across the whole set, not just the easy middle.

Also cover the non-model paths: the DICOM ingestion and windowing code, the thresholding and alerting logic, the fallback when the model is unavailable. Traditional code coverage is appropriate there and nowhere else.