Auto-Promoting the Latest Model Broke Retention Offers
A nightly Databricks job trains a churn model, registers it as main.ml.churn, and a second task calls the serving API to point the endpoint at the newest version. There is no validation gate. Last Tuesday, an ad-hoc experiment with a missing feature column registered version 47; AUC fell from 0.86 to 0.61 and the endpoint served it for six hours before an analyst noticed a 20% drop in retention offers accepted. The team uses MLflow tracking and Unity Catalog. They want automatic promotion but no repeat.
1. Design the job and promotion mechanism to prevent a bad model from reaching production, using aliases.
Read the worked answer
The failure is not that the model was bad; it is that registration and promotion were the same event. Registration should be cheap and reversible; promotion should be gated. Restructure the job into tasks: data quality checks, training, evaluation, a conditional promotion task, and an endpoint update task.
Training produces a model with a signature and input example. Evaluation uses mlflow.evaluate on a time-based holdout and checks absolute thresholds plus a comparison to the current champion, e.g. AUC >= 0.80 and not more than 0.02 below champion. Only if checks pass does the job register the version and assign the challenger alias. The endpoint should serve the champion alias, never a raw version number. Promotion moves the champion alias to the challenger version atomically, so rollback is another alias move. Missing-column failures are caught by signature validation before registration.
2. What should be monitored after promotion, and how would you roll back?
Read the worked answer
Enable inference tables on the endpoint so request payloads, predictions, and timestamps are logged. Monitor four things: request error rate and latency, feature null rates and schema mismatches, prediction score distribution versus the previous champion, and delayed label-based AUC once 30-day outcomes arrive. Alert on shifts such as a sudden change in score mean or a spike in nulls in a column that the model expects.
Rollback should be one operation: move the champion alias back to the last known-good version. Because the endpoint references the alias, traffic returns to the prior model without editing endpoint code. Keep the previous champion version and its training-set metadata for audit. After rollback, compare inference-table records from the bad interval to quantify impact and feed those examples into the next training and validation cycle.