Video analytics appliance bottlenecked before the NPU
An inspection appliance uses an Atlas 300I Duo inference card to analyze 16 RTSP 1080p streams with a YOLOv5s detector at 640x640. The requirement is 25 FPS per stream. The current pipeline uses OpenCV on the host to decode, resize, and normalize each frame, copies FP32 frames host-to-device, runs one static batch-1 OM model, then performs NMS on the CPU. End-to-end throughput is 11 FPS per stream, 310P utilization is 28%, and host CPU is 96% on 16 cores. Each FP32 frame is about 4.9 MB, so the 16 streams generate roughly 860 MB/s of host-to-device traffic at 11 FPS. DVPP and AIPP are unused. Frame drops occur in bursts when several streams emit keyframes together. The team proposes buying a second card.
1. Identify the dominant bottleneck and quantify why adding a second card would or would not help. Then redesign the pipeline stage by stage.
Read the worked answer
The NPU is not the bottleneck: 28% utilization with 96% host CPU points to decode and preprocessing on the host, plus per-frame copy and synchronization overhead. A second card would add inference capacity that the current pipeline cannot feed, so it would leave CPU saturation and bursty keyframe handling unchanged. The 860 MB/s host-to-device figure is not close to PCIe saturation, but the per-frame copy, FP32 format, and batch-1 launch/synchronization dominate latency.
Redesign: (1) move H.264/H.265 decode to DVPP hardware decode; (2) use VPC/DVPP resize and crop instead of OpenCV; (3) use AIPP for mean/std normalization, color conversion, and FP32-to-FP16 conversion so the device receives a compact YUV/RGB tensor; (4) batch frames across streams, for example batch 8, to amortize launch overhead; (5) keep tensors on device through inference and postprocessing, and run NMS on the NPU or a tightly bounded CPU stage; (6) use multiple ACL streams/threads so decode, inference, and postprocess overlap. Bursty keyframes should be absorbed by a bounded queue and frame dropping policy rather than unbounded buffering.
2. What capacity check and acceptance metrics would you use before deciding whether the existing card is sufficient?
Read the worked answer
Build a stage budget for the target 16 x 25 = 400 FPS: DVPP decode capacity, VPC resize capacity, AIPP throughput, NPU inference at the chosen batch, and NMS/postprocess throughput. Measure each stage independently with a synthetic 400 FPS input before full integration. If DVPP or VPC saturates first, reduce decode resolution or distribute streams; if NPU inference saturates, shrink the input, use a smaller model, or batch more efficiently; if postprocess saturates, move NMS to the device or reduce candidate counts.
Acceptance metrics: sustained 25 FPS per stream for at least one hour with zero unexplained drops; host CPU below 60%; 310P utilization in a healthy range, not pegged at 100%; end-to-end P99 latency within the inspection deadline; correct behavior when keyframes from multiple streams coincide; and detection mAP unchanged within an agreed tolerance after AIPP and FP16 conversion. Only if NPU inference remains the limiting stage after DVPP, AIPP, batching, and overlap should a second card be considered.