Computer-Vision Drift After a Resin Supplier Change
A medical-device manufacturer runs a computer-vision model on 12 injection-molding lines to detect surface defects. Each line produces about 40,000 units per shift. The model was validated at 0.7% false-negative rate and 1.1% false-positive rate on a held-out set from Supplier A. After Supplier B began supplying resin six weeks ago, quality engineers observed false negatives at 2.3% and false positives at 4.8%. Production wants to disable the manual sampling step to recover throughput; the model vendor says the issue is 'normal drift' and offers a retrain using the last six weeks of production images, which include inspector overrides. The quality director must decide whether to keep the model in-line, retrain immediately, or revert to manual inspection, and what controls to require.
1. Diagnose the most likely cause of the performance change and explain why immediate retraining on recent production images is risky.
Read the worked answer
The pattern, false negatives up from 0.7% to 2.3% and false positives up from 1.1% to 4.8% after a resin supplier change, suggests input distribution shift rather than ordinary noise. New resin can change surface gloss, color, texture, or defect appearance, so the model sees features outside its validation distribution. The simultaneous rise in both error types argues against a simple threshold problem.
Retraining on the last six weeks is risky because those images are not a clean labeled dataset. Inspector overrides may encode the inspectors' own adaptation, contain partial labels, and overrepresent defects the current model missed. Training on model-influenced production data can create a feedback loop and bake in the new supplier's quirks without stable ground truth. Six weeks may also not cover the full product mix or seasonal conditions. The correct first step is to freeze the in-line model, increase manual sampling, and build a stratified labeled dataset by supplier, line, and defect type before any retraining.
2. Recommend immediate and medium-term controls.
Read the worked answer
Immediate controls: revert to 100% manual sampling or at least a statistically justified higher rate until false negatives return below the validated limit; keep the model in shadow mode to collect predictions without releasing product; quarantine product made since the supplier change if traceability shows possible escape; notify quality and regulatory affairs if device safety or sterility is affected.
Medium-term controls: create a supplier-stratified golden test set with expert labels; retrain with balanced data and hold out an untouched validation set; validate per line and per supplier, not only in aggregate; add drift monitors on image embeddings, defect rates, and estimated false negatives from sampling; implement change control with pre-agreed acceptance criteria and rollback; require the vendor to provide model version, training-data summary, and known limitations.
Do not disable manual sampling to recover throughput while false negatives are more than three times baseline. The quality director should treat the vendor's 'normal drift' label as a hypothesis, not an explanation, and require evidence that the new resin's defect modes are represented in the retraining data.
3. Define acceptance criteria for returning to automated inspection.
Read the worked answer
Return only after a pre-registered validation shows all of the following:
- False-negative rate no worse than 0.8% overall and no worse than 1.0% in any supplier-by-line stratum, with a 95% upper confidence bound below 1.1%.
- False-positive rate no worse than 1.5% overall, with no stratum above 2.0%.
- The validation set contains at least 300 defect examples per stratum and spans both suppliers, all 12 lines, and the full defect taxonomy.
- Two weeks of shadow-mode operation show no safety-critical escapes and daily agreement with manual inspection above 99% on sampled units.
- Drift monitors are live, with alarms and a rollback runbook.
- Quality director and regulatory sign-off are documented.
If any stratum fails, keep manual sampling for that stratum. Aggregate metrics can hide a bad line, so the acceptance decision must be stratified. The director should also require a post-deployment check at 30 and 90 days.