Deployment, Inference, and Model Optimization Basics
Inference versus training
Inference is forward-pass only, latency- and throughput-sensitive, and often runs in reduced precision. Key metrics: latency per request, throughput (requests or tokens per second), and tail latency. A trap is optimizing average latency while ignoring p99, which drives user experience.
Optimization techniques
Know the toolkit: quantization (FP16, INT8, FP8), pruning, knowledge distillation, kernel fusion, and graph optimization. TensorRT builds an engine tuned to a specific GPU, precision, and input shape; engines are not portable across all configurations. Traps include assuming a TensorRT engine runs unchanged on different hardware or dynamic shapes without optimization profiles.
Batching strategies
Static batching groups fixed-size requests; dynamic batching groups requests arriving within a time window; continuous (in-flight) batching lets new requests join a running batch, which is important for LLM serving. A common trap is confusing continuous batching with simply increasing batch size, which raises latency.
Serving architecture
Triton Inference Server supports multiple frameworks and backends, dynamic batching, and model ensembles. NVIDIA NIM packages optimized models as containers with standard APIs. Know when to use each.
Worked rule of thumb
If latency is too high at low load, the problem is per-request cost (optimize the model or precision). If latency rises only under load, the problem is batching or capacity (tune batching and add replicas). Diagnose which regime you are in before changing anything.