Pilot-to-Production Gate for a Defect-Detection Vision Model
Vantage Plastics operates six injection-molding plants and is piloting a computer-vision system to detect short shots, flash, and sink marks on a high-volume automotive part. On the pilot line, the model achieved 98.2% recall and 0.8% false positives against 4,000 images curated by two retired quality inspectors. In a three-week production trial on a second line, recall fell to 71% and false positives rose to 6%. The second line uses a different camera angle, brighter LED lighting, and resin from a new supplier. The model was trained only on images from the pilot line and has no drift monitoring. The customer quality gate requires at least 95% recall before go-live, and a customer audit is scheduled in five weeks. The data science team proposes collecting 2,000 new images and fine-tuning for two weeks; quality wants to add manual inspection as a permanent backstop; operations fears throughput loss. No one has defined who owns model performance after handover.
1. Should the model pass the go-live gate? Justify the decision using the evidence rather than the pilot headline.
Read the worked answer
No. The 98.2% pilot recall is not evidence that the system meets the 95% gate in production. The pilot images were curated on one line and labelled by two inspectors without an adjudication process, so the headline figure is optimistic and its ground truth is uncertain. The second-line trial is the more relevant test: 71% recall and 6% false positives show distribution shift across camera angle, lighting, and resin. A model that misses roughly three in ten defects is not a 95%-recall system, and the false-positive rate would burden inspectors with rework.
The correct gate decision is a conditional fail: do not approve general rollout, but approve a time-boxed remediation path. Require revalidation on images drawn from each line and lighting condition, with a holdout that reflects production prevalence, not curated balance. Define acceptance as recall at or above 95% with a stated false-positive ceiling and a confidence interval, not a single point estimate. Keep manual inspection as the control until the evidence supports reduction. The customer audit in five weeks makes transparency essential: presenting a 71% trial as “nearly ready” would create quality and credibility risk. The gate should test fitness for the actual operating distribution, not effort expended.
2. Design the data collection and labelling plan needed to support a credible revalidation.
Read the worked answer
Start with a sampling frame tied to production reality: all six lines or at least every line and lighting/resin combination intended for deployment, sampled over full shifts to capture startup, steady state, and tool-wear effects. Include defect types in proportion to their operational importance and include hard negatives-visually similar but acceptable parts-so false positives are measured, not assumed away. Hold out entire lines and time periods for validation; random image splits leak near-duplicate frames and inflate results.
Ground truth needs a defensible protocol. Use at least two qualified inspectors per image, blind to model output, with a third adjudicator for disagreements. Record inter-rater agreement and resolve ambiguous cases against a physical or measurement standard where possible. Version the label set and document criteria for short shot, flash, and sink mark. Capture metadata-line, camera, lighting, resin lot, timestamp-so failures can be traced.
The 2,000-image proposal is a starting point, not a guarantee; estimate the needed sample from the recall confidence interval and defect prevalence. Fine-tuning on new images without fixing label quality would simply relearn noise. Budget time for relabelling the pilot set under the same protocol so training and validation are comparable.
3. What deployment and ownership controls should accompany any future go-live?
Read the worked answer
Run the model in shadow mode first: it scores production images but does not gate shipments, while inspectors remain the decision-makers. Compare model calls with inspection outcomes daily and publish recall and false-positive rates by line. Define a rollback trigger-for example, recall falling below the gate for two consecutive shifts or missingness in image metadata-and give quality, not the vendor, authority to invoke it.
Assign a named model owner with accountability for performance after handover, plus an operational owner for cameras, lighting, and data pipelines. Establish a monitoring set: input drift on image statistics, label drift, defect prevalence, and model confidence distributions. Agree a retraining cadence and an event-driven trigger for new resin, lighting changes, or camera replacement. Keep manual inspection as a defined backstop with staffing and throughput modelled, because the 6% false-positive rate will consume inspector time.
Document a model card covering intended use, training data, known failure modes, and acceptance thresholds. Add change control so a model update cannot reach the line without revalidation. The five-week audit deadline should be met with transparent trial evidence and a controlled plan, not a premature switch to automated release.