Bottle-cap inspection with a thin dataset and no reliable network
Tailspin's bottling line rejects about 0.4% of bottles for cap misalignment. Three inspectors per shift currently do the check visually. The team has labelled 1,180 images: 960 pass and 220 fail, captured on two cameras with noticeably different lighting. The line runs at 600 bottles/minute, so inference must finish in under 100 ms per frame. The plant's uplink drops several times per day. A prototype using a cloud image classification model returns in 400-900 ms per call. Recall on defective bottles must exceed 0.95, and false rejects above 2% stop the line for investigation.
1. Choose the inference architecture and defend it against the alternative.
Read the worked answer
Decision: edge inference with a cloud second-opinion tier.
- Latency and connectivity disqualify per-frame cloud calls. 400-900 ms per call against a 100 ms budget is a 4-9x miss, and the uplink outages would halt inspection entirely. No amount of prompt or region tuning closes that gap; the round trip is the floor.
- Concretely: train a compact Custom Vision classification model (compact domain), export it as ONNX or a Docker container, and run it on a gateway device next to the line. Compact models trade a few points of accuracy for size and speed - acceptable because the second tier catches the rest.
- Second tier: frames whose predicted probability falls in the uncertain band (e.g. 0.40-0.70) are buffered locally and sent to the higher-accuracy cloud model when connectivity returns, for relabelling and audit - not for real-time rejection.
- Why not Azure AI Vision Image Analysis custom models? They are cloud-only with no edge export, so they inherit the same latency and outage problem.
Open risk to name: 220 defect images is thin. Budget for a data-collection phase, not just a model-selection decision.
2. How should the team evaluate and tune the model, given a 0.4% defect rate?
Read the worked answer
Accuracy is the wrong metric and will actively mislead.
- A model that predicts 'pass' for every bottle scores 99.6% accuracy on this distribution and never catches a defect. Reporting accuracy will make a useless model look excellent.
- Report precision and recall on the defect class, plus the confusion matrix. Recall must exceed 0.95 by requirement; false rejects must stay under 2% of all bottles.
- Tune the decision threshold, don't accept 0.5. Lowering the defect threshold raises recall and false rejects together. Sweep the threshold and pick the operating point that satisfies recall ≥ 0.95 with false rejects ≤ 2%; if no point satisfies both, the model or the data needs work, and that is a finding to escalate rather than paper over.
- Address the imbalance: class weighting, targeted augmentation of defect images (rotation, brightness, blur to mimic motion), and per-camera breakdown of recall - one camera's lighting may be masking defects.
- Hold out by time and by camera, not randomly: a random split leaks near-duplicate frames from the same bottle into both sets and inflates results.
3. How do you handle the two cameras, and what does the retraining loop look like in production?
Read the worked answer
Cameras: one model now, per-camera monitoring, per-camera models only if evidence demands it.
- Splitting 220 defect images across two cameras gives ~110 each - too few for two reliable models. Train one model, include both cameras, and use brightness/contrast augmentation to reduce lighting sensitivity.
- Track recall and false-reject rate per camera in production. If camera B degrades while A holds, that is evidence for splitting, or more likely for fixing the lighting rather than the model.
Retraining loop:
- Human reviewers confirm every reject and every frame in the uncertain band; those labels are the training signal.
- Drift monitors: distribution of predicted probabilities, and the share of frames in the uncertain band - a rising trend precedes a recall drop.
- Deploy safely: version the exported model, shadow-run the candidate on recorded frames, compare recall/false-reject against the incumbent before promoting, and keep the previous container for instant rollback.
- Never retrain-and-deploy automatically on a live line - a silent recall regression is a quality escape, not just a bad metric.