AI Observability
AI Observability
The AI Observability domain tests whether you can detect, diagnose, and act on production degradation for predictive models, GenAI apps, and agents. DataRobot monitoring concepts include data drift, target drift, accuracy, service health, alerts, and challenger models. The exam emphasizes interpretation: an alert is a signal to investigate, not a verdict.
Core rules
- Establish a baseline from training or a known-good production window; without it, drift scores are meaningless.
- Separate data drift (input distribution), target drift (outcome rate), and concept drift (input-output relationship).
- Track accuracy directly when labels arrive; when labels are delayed, use proxy metrics and label latency explicitly.
- Segment metrics by region, channel, product, and protected group; aggregate health can hide local failure.
- Set alert thresholds from historical variation and business cost, not arbitrary defaults.
- Maintain challenger models and define retraining triggers in advance.
- For GenAI and agents, trace retrieval, prompts, tool calls, token cost, latency, refusals, and safety events.
Worked example
A loan-default model alerts on income drift: PSI 0.25, above the 0.20 threshold. Accuracy is still 0.81 versus 0.82 baseline, so the team does not retrain immediately. Segmentation shows the drift is concentrated in one new region where application volume grew. The team adds a region-specific challenger, monitors it for four weeks, and finds its AUC is 0.03 higher in that segment. They promote the challenger for that region only. Reasoning: drift alone did not prove degradation; segmentation and a challenger test connected the signal to a decision.
Traps
- Retraining on every drift alert; drift can be benign or even favorable.
- Monitoring only aggregate accuracy and missing a failing segment.
- Ignoring delayed labels and declaring the model healthy because no accuracy alert fired.
- Setting thresholds so tight that alert fatigue causes real incidents to be ignored.
- No baseline, owner, or runbook, so an alert has no defined action.
- For agents, logging only final answers and not tool calls, retrieval, or cost.