INT8 Quantization of a Visual Defect Detector
A semiconductor fab uses an OpenVINO-based visual defect detector on 12 inspection cameras. The FP32 model runs at 34 ms per image on an Intel Core i7 edge box, but the line requires under 20 ms at 30 FPS with two cameras per box. The team runs NNCF post-training quantization to INT8 with 300 calibration images. [email protected] falls from 0.86 to 0.79, and recall for the smallest defect class falls from 0.84 to 0.71. Only 1,200 labeled images exist, with 60 positives for that class. The fab cannot accept more than a 2-point mAP drop and cannot add cameras. Options discussed: more calibration images, per-channel quantization, mixed INT8/FP32, quantization-aware training, or a smaller model. The edge box has an integrated GPU, but the current pipeline is CPU-only.
1. Sequence the optimization steps. What should the team try before committing to quantization-aware training?
Read the worked answer
Before QAT, exhaust cheaper levers. First validate the calibration data: 300 images may under-represent the smallest defect class. Build a stratified calibration set of 500 to 1,000 images that covers all defect types, lighting conditions, and camera models, but keep a separate held-out test set. Second, confirm preprocessing parity between training and OpenVINO inference; a resize or normalization difference often explains more accuracy loss than quantization. Third, enable per-channel weight quantization and asymmetric activations where supported, and use NNCF to ignore the first and last layers or the detection head if those are the sensitive scopes. Fourth, try mixed precision, keeping the detection head in FP32 while quantizing the backbone. Fifth, measure whether the integrated GPU via the OpenVINO GPU plugin meets the 20 ms target without INT8; offload may be simpler than QAT. Only if these fail should the team invest in QAT, which requires a training pipeline and more labeled data.
2. Design the calibration and evaluation protocol so the team does not overfit to the 1,200 labeled images.
Read the worked answer
Split by time and camera, not randomly, so near-duplicate frames from the same lot do not leak across splits. Use roughly 70% for calibration and 30% as a locked test set, and keep an additional 10% as a final audit set that is touched once. Calibration images should be selected to cover the rare class and the production distribution; do not tune the calibration set against the test set. Evaluate with [email protected], per-class recall, and a small-defect recall slice, and report confidence intervals because 60 positives is a small sample. Run the FP32 model on the same test set to establish the baseline, then compare each quantization variant. Track false negatives per shift, since a missed defect has asymmetric cost. Pre-register the acceptance threshold and the exact test set before running experiments.
3. Recommend a go/no-go criterion and a fallback if INT8 still misses the threshold.
Read the worked answer
Go only if the locked test set shows at most a 2-point mAP drop, small-defect recall within 3 points of FP32, and p95 latency under 20 ms at two cameras per box. Because 60 positives give wide confidence intervals, require the result to hold on the audit set and on a week of shadow traffic. If INT8 fails, fall back in this order: mixed precision with the detection head in FP32; GPU offload via the OpenVINO GPU plugin; a smaller model retrained with distillation from the FP32 teacher; and only then QAT. If none meets the threshold, the correct decision is to escalate the line-speed constraint rather than silently accept a safety-critical recall loss. Document the rejected variants and the rationale for the audit trail.