Deployment, Serving, and Orchestration
Domain 5 covers packaging, serving, orchestration, and safe rollout. The exam expects versioned artifacts, reproducible deployment, health checks, autoscaling, and rollback.
Serving patterns
- Build the model artifact separately from the serving image; pin image digests and model versions.
- OpenVINO Model Server exposes KServe-compatible APIs, dynamic batching, and a model repository; it is a strong fit for Intel CPU inference.
- Kubernetes deployments need readiness/liveness probes, resource requests and limits, and horizontal pod autoscaling based on the real bottleneck.
- Orchestrators such as Kubeflow or Airflow schedule training and batch inference DAGs; they should be idempotent and retry-safe.
Rollout rules
Use shadow or canary deployment. Route a small percentage of traffic, compare error rate and p95 latency against the baseline, and promote only if both quality and SLOs hold. Keep the previous artifact deployable for one-step rollback.
Worked example
Deploy a new OVMS model to Kubernetes with three replicas. Send 10% of traffic to the canary for 30 minutes. Baseline p95 is 150 ms and error rate 0.5%; canary p95 is 180 ms and error rate 0.6%. Reasoning: latency is within a 200 ms SLO but error rate is above the 0.5% threshold; hold the rollout, inspect logs for preprocessing mismatches, and roll back if the issue is model-related. Do not promote on average latency alone.
Traps
- Using the latest tag and losing reproducibility.
- No readiness probe, so traffic reaches a loading model.
- Autoscaling on CPU when the bottleneck is memory or queue depth.
- Skipping rollback planning.
- Letting preprocessing drift between training and serving.