Low-latency fraud model releases on AKS
A payments company runs a fraud model on Azure Kubernetes Service. Peak traffic is 9,000 transactions/sec, the p99 latency budget is 45 ms, and the model must be refreshed daily. The current process bakes the model file into a new container image, pushes it to a registry, and runs kubectl set image. A rollout takes 22 minutes because readiness probes wait on a 15-minute warm-up that loads a 2.8 GB feature-embedding table. Rollback requires rebuilding the previous image and takes about 40 minutes. Last week a bad model raised false declines by 18% for 35 minutes before rollback. The team has a model registry and 12 inference pods, and wants canary releases, near-instant rollback, and no cold-start latency spikes. The feature-embedding table changes weekly and is copied into the image at build time.
1. Recommend a deployment and model-serving architecture that removes slow image rebuilds and cold-start spikes.
Read the worked answer
Separate the model artifact from the serving image. Build one generic inference image per runtime and pull the versioned model from the registry at pod startup via an init container, or mount it from a read-only volume. That removes 22-minute image builds for model-only changes and lets rollback select the previous registry version. For the 2.8 GB embedding table, preload during startup into a memory-mapped file or local SSD cache; use a startup probe so readiness only passes after warm-up. Keep a stable deployment warm and switch traffic at the service or ingress layer. Reject baking the model into the image because it couples model release to container build and makes rollback slow. Version the embedding table with the model and test parity when it changes weekly.
2. Design the canary release and rollback process, including the signals that decide promotion.
Read the worked answer
Use two deployments, stable and canary, behind a weighted router. Send 1-5% of traffic to canary, watch false-decline rate, p99 latency, error rate, and score distribution, and auto-promote or roll back. Rollback is a traffic switch to the stable deployment, not an image rebuild. Keep the previous model loaded in the stable deployment or in pre-warmed spare pods. Use a service mesh or ingress with weighted routing; if none exists, use separate services and adjust selectors cautiously. Add a PodDisruptionBudget and autoscaling on concurrency. The decision signals must include business false declines, not only infrastructure metrics, because the last incident was detected by an 18% rise in false declines rather than by pod health.
3. What latency and capacity considerations should be validated before enabling canary releases?
Read the worked answer
Validate per-pod throughput with load tests: 12 pods must handle 9,000 TPS at p99 45 ms with headroom for one pod failure. Warm-up must not count against readiness, so use startup probes and a preloaded cache. Node-local cache for the embedding table and a preloaded model reduce cold-start latency. If the table changes weekly, version it and run a parity check between old and new tables. Canary routing adds some overhead; measure it. Keep a warm spare capacity buffer so rollback and canary both have capacity. Document the rollback trigger thresholds and test the rollback path before relying on it.