Model Monitoring and Maintenance
What the exam tests
This domain tests how you detect degradation after deployment and decide when to retrain. Expect questions on Lakehouse Monitoring, inference tables, drift metrics, ground-truth joins, alerting, and the difference between data drift, concept drift, and model quality loss.
Core rules
- Lakehouse Monitoring creates profile and drift metrics for tables. A monitor compares a baseline table or time window with current data and can publish metrics to Delta tables for querying and alerting.
- Inference tables provide the serving-side data needed for monitoring. Without request and response capture, you cannot compute production feature distributions or prediction distributions.
- Model quality monitoring requires labels. Join delayed ground truth to inference records by request ID or timestamp, then compute accuracy, precision, recall, RMSE, or business metrics.
- Data drift means input distributions change. Concept drift means the relationship between inputs and target changes. A model can show data drift without quality loss, or quality loss without obvious input drift.
- Monitor freshness, completeness, and volume as well as statistical drift. A silent pipeline failure can look like stable distributions if no rows arrive.
- Alerts should trigger an action: investigate, roll back an alias, or retrain with newer data.
Worked example
A credit model writes requests and scores to an inference table. A Lakehouse monitor compares current applicant features with a baseline month. After six weeks, the monitor shows strong drift in income_band and loan_term. The team joins approved/declined outcomes that arrive 30 days later and finds recall has fallen. They investigate a changed application flow, refresh the baseline after validating the new population, retrain on recent labeled data, and promote through the challenger alias. Reasoning: drift alone is a signal, not proof of failure; the label join shows actual quality loss and justifies retraining.
Traps that cost marks
- Monitoring only feature drift and never joining labels, so actual model quality is unknown.
- Comparing production data to the wrong baseline, such as a stale training sample that no longer represents the deployed population.
- Forgetting timestamp keys, which makes drift windows and label joins impossible.
- Alerting on every statistical fluctuation without thresholds tied to business impact.
- Assuming retraining always fixes concept drift when the labeling process itself has changed.
- Leaving inference tables disabled and expecting serving metrics to appear automatically.
Key points
- Lakehouse Monitoring computes profile and drift metrics against a baseline.
- Inference tables supply production request/response data for monitoring.
- Model quality needs ground-truth labels joined to inference records.
- Data drift, concept drift, and quality loss are distinct signals.
- Alerts must lead to investigation, rollback, or retraining.