ML Operations: Registry, Promotion, Orchestration, and Monitoring
The ML Operations domain tests the lifecycle after a model trains: versioning, promotion, automation, monitoring, and governance. Questions often describe a model that works but is promoted by overwriting a file or a hard-coded version, then ask what control would have prevented the incident.
Registry semantics and promotion
In Unity Catalog a registered model has a three-level name, catalog.schema.model, and immutable versions. Use aliases such as champion and challenger; Unity Catalog model versions do not use stages, so stage-based promotion is a legacy workspace pattern. Moving an alias is atomic and does not require changing a serving endpoint that references models:/catalog.schema.model@champion. Grant USE CATALOG, USE SCHEMA, and EXECUTE on the model to the job or serving service principal; model ownership alone is not enough for a different principal.
Automate with gates, not timers
A multi-task Databricks job can train, evaluate, run a conditional task, and request approval. Databricks Asset Bundles plus Git-backed Repos make the job definition reviewable. The promotion gate should compare a challenger to the current champion on a time-based holdout using the same metric and threshold, then set the alias only if the gate passes. Log the evaluation as a run so the decision is auditable.
Monitor data and model quality
Enable inference tables on serving endpoints to capture requests and responses. Lakehouse Monitoring can profile the inference table, compare it to a baseline training table, and compute drift and performance metrics when ground truth is joined later. Alert on drift, but also on missing ground truth, latency, and error rate. Retraining can be scheduled or drift-triggered; either way it should produce a challenger, not overwrite the champion.
Worked example
A daily job trains a challenger, evaluates AUC on the most recent 90 days, and compares it with the champion's AUC on the same window. If the challenger improves by at least 0.01, a conditional task creates an approval request; on approval, a task calls set_registered_model_alias to point champion at the new version. The serving endpoint references the alias, so no endpoint edit is needed. Reasoning: same-window comparison controls for drift, the threshold prevents noise-driven promotion, and the approval task satisfies governance.
Traps that cost marks
- Using stages in Unity Catalog instead of aliases.
- Moving the champion alias before the gate finishes.
- Monitoring only request volume, not input drift or delayed performance.
- Forgetting that a job's service principal needs model privileges.
- Retraining on a schedule without a baseline or rollback alias.