Model Monitoring, Evaluation, and Responsible AI
This domain tests how watsonx.governance records, evaluates and monitors models, and the metrics used at each stage.
Governance lifecycle
Use case → model inventory → factsheet → risk assessment → pre-deployment evaluation → production monitoring. Factsheets collect model details, training data, intended use and lineage.
What monitoring measures
- Fairness - group-level outcomes and disparate-impact screening.
- Drift - data drift in features, model drift in predictions; requires payload logging.
- Quality - accuracy or ROC over time against labeled feedback.
- Explainability - SHAP, LIME and contrastive explanations.
Generative AI guardrails
Hate, abuse and profanity (HAP) detection and PII detection, plus groundedness checks for retrieval-augmented answers.
Metrics by task
Classification: accuracy, precision, recall, F1, ROC AUC, confusion matrix. Regression: RMSE, MAE, R². Generative: ROUGE, BLEU, perplexity, exact match, plus human or model-judged fidelity.
Worked example
A deployed model's F1 drops after a pricing change. Reasoning: compare production feature distributions against training data first. If drift is present, refresh training data and retrain; retraining immediately hides the cause and may repeat it.
Traps
Monitoring without enabling payload logging; treating fairness as a single number; comparing metrics computed at different thresholds; assuming high ROUGE means factual correctness; recording a factsheet only once and never updating it.