Model Management
Domain 6: Model Management
Model Management covers the governance of models across their lifecycle. CAP questions address inventory, ownership, validation, monitoring, retraining, version control, documentation, audit, and retirement.
Core rules and concepts
- Maintain a model inventory with purpose, owner, tier or risk rating, data inputs, version, deployment location, and status.
- Assign clear ownership: business owner, model owner, validator, and technical operator. Separate development and validation where risk warrants.
- Use version control for code, data snapshots, parameters, and documentation; reproduce any production score from a known version.
- Monitor performance and stability: outcome metrics where labels arrive, proxy metrics where they do not, and drift measures such as PSI or KS on features and scores.
- Define retraining triggers in advance: scheduled cadence, performance degradation thresholds, drift thresholds, or business change. Retain pre-retraining validation.
- Govern changes through approval, testing, documentation, and communication. Avoid undocumented manual overrides.
- Retire models deliberately: archive artifacts, revoke access, update downstream systems, and record the rationale.
Worked example
A monthly propensity model is monitored with PSI on key features and score distribution, plus AUC when outcomes mature. The governance plan sets triggers: PSI above 0.2 on any top feature, AUC drop of more than 5% from validation, or a major pricing change. Retraining is not automatic; the model owner prepares a change request with validation evidence, the independent validator reviews it, and the business owner approves. The new version is shadow-scored for one cycle before promotion. If performance degrades without a clear cause, the team can roll back to the prior version. Documentation records the trigger, evidence, approver, and effective date.
Traps that cost marks
- Treating a deployed model as set and forget with no monitoring or owner.
- Retraining on every schedule without validating that the new model is better.
- Losing reproducibility through untracked data, code, or manual changes.
- Monitoring only statistical drift and ignoring business outcomes or data quality.
- Allowing undocumented overrides or shadow models outside governance.
- Retiring a model without updating dependent systems or archiving evidence.