Model Optimization and Inference with Intel Toolkits
Domain 4 tests model compression and inference optimization with Intel toolkits: Intel Neural Compressor, OpenVINO, and framework extensions. You must improve latency or throughput while measuring accuracy on target hardware.
Optimization rules
- Establish an FP32 baseline and a representative calibration set before compressing.
- Post-training quantization (PTQ) is fast but sensitive to outliers; quantization-aware training (QAT) costs more and often preserves accuracy.
- Pruning removes weights or channels; distillation transfers a teacher’s behavior to a smaller student. Both need fine-tuning and accuracy checks.
- OpenVINO converts models to IR and runs them with CPU/GPU plugins; AUTO selects devices, HETERO splits unsupported layers, and MULTI increases throughput.
Worked example
A ResNet-style classifier runs at 120 ms per image on an Intel CPU. PTQ to INT8 with 300 representative images gives 35 ms and a 0.4-point top-1 drop. Reasoning: 0.4 points may be acceptable if the business threshold is 1 point, but check per-class recall and the calibration distribution. Inspect the IR for layers that fell back to FP32; if the drop exceeds 1 point, try QAT or mixed precision. Report p95 latency and throughput at the production batch size, not a single warm run.
Traps
- Calibrating on the training set or a tiny, unrepresentative sample.
- Comparing latency on a laptop with the production server.
- Forgetting preprocessing and postprocessing costs.
- Assuming INT8 is always faster; memory-bound or unsupported operators can regress.
- Over-pruning without re-validation.