AI Application Development and Deployment
Domain scope
This domain tests the path from a trained checkpoint to a reliable service: optimization, serving, capacity planning, and monitoring. Expect questions on latency versus throughput, quantization trade-offs, and rollout safety.
Core rules and concepts
- Separate preprocessing, inference, and postprocessing; profile each stage. End-to-end p95/p99 latency matters more than mean latency.
- Batching raises throughput but adds queueing delay. Dynamic batching trades latency for utilization.
- Optimization: operator fusion and graph compilation reduce launch overhead; INT8 quantization needs a calibration set; pruning and distillation reduce compute but require retraining or fine-tuning.
- Serving: expose health/readiness endpoints, use a model registry, pin versions, and roll out canary or shadow traffic before full promotion.
- Monitoring: track error rate, latency, saturation, input drift, and label delay. Drift alerts should trigger investigation, not automatic retraining.
Worked example
An image service must sustain 200 images/s. The model processes a batch of 8 in 40 ms. Per device, 1/0.040 = 25 batches/s, or 200 images/s at 100% utilization. That leaves no headroom, so at a 70% target utilization one device is insufficient: required devices = 200/(200×0.70) = 1.43, round up to 2. If preprocessing adds 10 ms end-to-end, each batch takes 50 ms, capacity falls to 160 images/s per device, and three devices are needed at 70% utilization.
Traps that cost marks
- Quoting batch-1 throughput as service capacity while ignoring batching and queueing.
- Reporting average latency when tail latency drives user experience.
- Quantizing with unrepresentative calibration data and then discovering accuracy loss in production.
- Deploying without readiness checks, version pinning, or drift monitoring.