Model Development, Validation, and Documentation
What this domain tests
Development, validation, and documentation questions test independent challenge, evidence quality, and whether documentation supports decisions after deployment.
Core rules
- Validation is independent of development and covers conceptual soundness, data quality, methodology, implementation, outcomes, sensitivity, and benchmarking.
- Effective challenge means a qualified party can question assumptions, test alternatives, and impose conditions.
- Documentation should state purpose, scope, owners, data lineage, methodology, assumptions, limitations, performance, fairness testing, approvals, and change history.
- Validation reports should distinguish limitations, compensating controls, and conditions for approval.
- Model risk management expects ongoing validation, not a one-time approval; material changes trigger revalidation.
- A model card is a communication artifact, not a substitute for a validation report.
Worked example
A probability-of-default model reports AUC 0.82 in development. Independent validation finds poor calibration in the low-risk grade and performance sensitivity to unemployment. The validator does not reject on AUC alone; it requires recalibration, a macro-sensitivity test, and a condition that the model not be used for pricing until calibration is fixed.
Traps
Using training metrics as validation evidence; documenting after go-live; treating validation as only performance testing; and accepting vendor validation without local evidence.