Model deployment and serving
Model deployment and serving tests how you package a registered model into a reliable endpoint with correct authentication, scaling, traffic control, and rollback.
Core rules
- Managed online endpoints are for real-time inference; batch endpoints are for file or large-table scoring; Kubernetes online endpoints are for AKS-specific control.
- A deployment needs a model, environment, scoring script with init and run, and an inference configuration. Test the scoring script locally before deployment.
- Use key or token auth deliberately; prefer Microsoft Entra ID and managed identity for service-to-service calls. Configure autoscale, resource requests/limits, and health probes.
- Use blue/green or canary traffic splitting. Send a small percentage to the new deployment, compare error rate and latency, then shift traffic or roll back.
Worked example
A sentiment model v2 is ready. The team deploys v2 to a second deployment on the same managed online endpoint with 10% traffic. Application Insights shows v2 p95 latency at 180 ms and error rate 0.4%, close to v1. After a smoke test and two hours of comparison, traffic moves to 50%, then 100%. If v2 error rate exceeded 2%, the team would set v2 traffic to 0 and keep v1 serving.
Traps
- Deploying without a local scoring-script test.
- Mismatched request schema between client and scoring script.
- No resource limits, causing noisy-neighbor failures.
- Treating endpoint keys as long-lived production secrets.
- Skipping a canary and switching 100% at once.