Model Performance, Monitoring, and Drift
What this domain tests
Performance and monitoring questions test metric selection, drift diagnosis, and response. You need to know what changed and why it matters.
Core rules
- Choose metrics tied to the decision: ranking metrics for prioritization, calibration for probabilities, and cost-weighted measures for thresholds.
- Data drift is a change in inputs; concept drift is a change in the input-output relationship; label drift is a change in outcome prevalence.
- Delayed labels require proxy monitoring, such as input stability, approval rates, and early outcomes.
- Stability metrics such as PSI or KS compare distributions; thresholds should be set from historical variation and risk appetite.
- Monitoring must include data pipelines, overrides, vendor changes, challenger models, and incident escalation.
Worked example
A credit model shows PSI 0.25 on income, an 8-point rise in approval rate, and rising early delinquency. PSI alone suggests input drift; the approval and delinquency signals suggest the threshold or population changed. The analyst should trace the pipeline, compare champion and challenger, and pause automated approvals if harm is plausible.
Traps
Monitoring only AUC; retraining without validation; treating every alert as model failure; and ignoring upstream data or process changes.