Edge Video Analytics on Atlas 300I Pro Cards
A city surveillance integrator must run four models (person detection, helmet detection, plate OCR, loitering) on 64 incoming 1080p25 camera streams at the edge. Each channel is sampled at 2 fps per model, so each model must handle 128 inferences per second, and end-to-end latency must stay under 40 ms. The box is an Atlas 800 edge server with four Atlas 300I Pro cards (8 TOPS INT8 each), CANN 6.0, and four 210 MB FP32 ONNX models. A prototype using CPU decoding and FP32 models reaches only 19 fps aggregate, saturates three CPU cores, and misses the latency target. DVPP is unused, and every model was converted with ATC at batch 1 in FP32. The engineer proposes raising the batch size to 32 and buying a fifth card. No additional hardware budget is approved.
1. Diagnose the bottleneck. Is batching to 32 and adding a fifth card the right response?
Read the worked answer
The arithmetic points away from the NPU. Decoding 64 streams at 25 fps requires 1,600 frames per second before any inference. Software 1080p decoding runs at roughly 100–200 fps per CPU core, so 8–16 cores are needed; three saturated cores confirm that decode and preprocessing are the bottleneck, not the 32 TOPS of INT8 compute.
Batching to 32 is also wrong for latency. At 2 fps per stream, filling a 32-frame batch from one stream takes 16 seconds. Even batching across all streams at 128 fps takes 250 ms, far beyond the 40 ms budget. A fifth card would add capacity that the pipeline cannot feed.
The correct response is to offload decode and resize to DVPP, use AIPP for color conversion and normalization, convert models with ATC to INT8, keep batch 1 per stream, and run multiple model instances in a pipelined scheduler. Profile again before purchasing anything.
2. Describe the ATC conversion and accuracy-validation steps for INT8 deployment.
Read the worked answer
ATC converts the ONNX model to an .om offline model. Specify the correct --soc_version, fixed --input_shape, --precision_mode, and an AIPP configuration matching DVPP output. For INT8, provide a calibration dataset drawn from the actual camera scenes, covering day, night, rain, and plate angles; a calibration set from one scene will skew the scale factors.
After conversion, compare per-model accuracy against the FP32 baseline on a held-out set: mAP for detection, character accuracy for OCR, and event precision and recall for loitering. Set an acceptance threshold per model, for example no more than a one-point drop. If plate OCR degrades, keep it in FP16 while the detectors use INT8. Verify that AIPP color format and mean/scale match training preprocessing; a mismatch silently damages accuracy. Re-run the latency test with msprof and check that DVPP output feeds AIPP without a CPU copy.
3. Define the acceptance test that proves the box can meet the 40 ms latency target without new hardware.
Read the worked answer
Compute the required concurrency. Four models × 128 inferences per second is 512 inferences per second. With a 40 ms budget, about 512 × 0.04 ≈ 20.5 inferences must be in flight at any moment, so the scheduler needs enough parallel instances and queues to keep the devices fed.
The acceptance test should run all 64 streams simultaneously for at least one hour and measure p50 and p99 end-to-end latency, not average. Pass criteria: p99 below 40 ms, aggregate throughput above 512 inferences per second, CPU utilization below 50%, no frame drops caused by queue overflow, and accuracy within the agreed drop on a scene-representative sample. Monitor per-model latency to find a dominant model; if OCR dominates, allocate it more instances rather than adding a card. Only if p99 still fails after DVPP, AIPP, INT8, and pipelining should hardware be reconsidered.