Hairline crack detection: classification model on a 180-image defect set
An automotive supplier inspects 4,000 parts per shift with a fixed 1280x720 camera. QA labeled 180 images across six defect classes; the rarest class has 9 examples, and hairline cracks occupy roughly 20x8 pixels. The current Azure AI Custom Vision multi-class classification model reports 94% overall accuracy but only 71% recall on the rare class, and it flags parts as defective without indicating where, so line operators cannot tell what was wrong. The line must run disconnected from the internet for hours at a time, and end-to-end inspection must complete in under 300 ms per part. Retraining happens monthly with newly labeled images from the QA team.
1. Explain why the current model's headline accuracy is misleading and what shape of model the problem actually requires.
Read the worked answer
Two separate problems, only one of which is fixable by tuning.
1. The metric is lying. With six classes where one has 9 examples, a classifier that predicts the majority class for everything can score high "overall accuracy" while never detecting the rare defect. 94% overall with 71% recall on the rare class is exactly the signature of a model that has learned the class prior. Report per-class recall and precision, plus a confusion matrix, and set the acceptance criterion on the rare class - that is the one that costs money.
2. The task framing is wrong. Classification assigns one label to the whole image and produces no localization. The operators need a bounding box: "defective, and here." Also, a 20x8 crack in a 1280x720 frame is about 0.02% of the pixels; at typical training resolutions the crack is one or two pixels wide and effectively disappears. Resizing is destroying the evidence.
What to build.
- Switch to object detection so the output is class + bounding box + confidence.
- Tile the frame into overlapping crops (e.g., a 3x4 grid with 20% overlap) at native resolution and run detection on the crops, then merge boxes back to full-frame coordinates. A coarse-to-fine cascade is the cheaper variant: a low-resolution pass to find candidate regions, then a high-resolution crop only for those regions.
- Augment to compensate for 180 images: flips, small rotations, brightness/contrast, blur, and crops that preserve the defect (not aggressive downscaling).
- Collect more of the rare class before anything else; 9 examples cannot support a reliable detector regardless of architecture. Every false negative on a crack is a safety and warranty event, not a nuisance.
Judgement point. Do not try to rescue this by raising the classifier's resolution alone. Localization is a business requirement, and detection also gives operators the feedback loop that improves labeling quality.
2. Recommend how to run inference at the edge within 300 ms, and name the tradeoffs.
Read the worked answer
Options, in the order I would evaluate them:
- Export the trained model as a container and run it on the line. Custom Vision supports exporting detection models as Docker containers for offline inference on x86 or ARM devices, and as ONNX/TensorFlow artefacts. This meets the disconnected requirement outright: no network dependency, model version pinned on the device, predictable latency. Tradeoffs: container image and runtime must match the device architecture; no automatic model updates (you need a deployment/versioning mechanism); the offline device is a maintenance surface.
- Train and deploy through Azure Machine Learning to an edge target (IoT Edge / ONNX Runtime). More control over the model family, quantization, and hardware acceleration, at the cost of much more engineering. Use when the exported model cannot hit the latency target.
- Cloud inference (Azure AI Vision / a hosted endpoint). Rejected by the connectivity requirement. Not a candidate.
Latency maths for the tiling design. If detection takes ~15 ms per tile on a CPU device and you run 12 tiles per part, that is ~180 ms plus capture and merge, which fits under 300 ms but with little headroom. Two levers: reduce tile count with a coarse-to-fine cascade (1 full-frame pass at low resolution, then high resolution only on 1–3 candidate regions), or move to a GPU/accelerated edge device. Measure on the actual line hardware with the real camera pipeline, not on a developer laptop.
Judgement point. Do not over-automate the retraining loop yet. With monthly manual labels from QA, a versioned model artefact, a shadow-mode period on the device, and a documented rollback are worth more than an automated pipeline. Also verify the current lifecycle/retirement status of whichever Custom Vision capability you depend on before locking the architecture in, and keep the training data and export path portable in case you must move.
3. How would you handle the rare class and choose the decision threshold, given the costs on both sides?
Read the worked answer
Data first. Nine examples is not enough to learn a rare, high-consequence class. Before touching thresholds, run a targeted collection drive on the rare class - including captures at slightly different angles, lighting, and part positions - and use augmentation (rotation, brightness, contrast, blur, partial occlusion) to build invariance. Synthetic generation of cracks helps as augmentation, not as a substitute for real confirmed defects. Track a held-out set of confirmed rare-class images that is never trained on, or every metric you report is optimistic.
Threshold as an economic decision, not a default.
- Miss (false negative): a cracked part proceeds downstream. Use a conservative $400 per escape (scrap, rework, warranty exposure) and note the safety multiplier.
- False alarm (false positive): line stop and manual re-inspection. At 4,000 parts/shift over roughly 480 minutes, the line runs ~8 parts/minute, so a 90-second stop costs about 12 parts of throughput plus operator time - call it tens of dollars, an order of magnitude cheaper than an escape.
- That asymmetry says: set the operating threshold low (favor recall), and accept the false-positive rate, as long as re-inspection capacity can absorb it. Compute expected cost per part at candidate thresholds on the held-out set and pick the minimum; don't pick 0.5 because it is tidy.
Three-band routing instead of one threshold.
- Above the high threshold: auto-reject, operator confirms disposition.
- Middle band: route to operator adjudication with the box and confidence shown.
- Below: auto-accept.
Judgement point. Report precision/recall at the chosen threshold per class, and re-check monthly as new labels arrive - the 94% headline should never appear in a status report again. Also monitor the false-positive rate on the line: if operators start dismissing alerts reflexively, the threshold has drifted too low and the effective recall is worse than the measured one.