Regulatory Reporting When a Model Change Slips Past the Compliance Calendar
A bank uses an ML model to generate inputs for a regulatory capital calculation. The model is classified as 'Tier 2' under the bank's internal model risk policy, which requires annual validation and notification to the regulator only for 'material' changes. In Q1, the data science team retrained the model on two additional years of data and switched from a linear to a gradient-boosted approach. The change was logged in the engineering changelog but not routed to model risk management, because the team judged it a 'routine retraining.' The model's outputs shifted the capital input by 4.2% on average, with some segments moving 9%. The regulator's next scheduled review is in five months. The model risk policy defines 'material' qualitatively ('significant change in methodology or output') with no numeric threshold. Internal audit finds the gap.
1. Was this change 'material' under the policy? Reason through the ambiguity.
Read the worked answer
The policy is qualitatively defined and therefore ambiguous, which is itself a finding. Applying it:
- Methodology change: linear → gradient-boosted is a fundamental change in model form, not a parameter refresh. On its face this is 'significant change in methodology.'
- Output change: 4.2% average, 9% in some segments, on a regulatory capital input. Capital inputs are high-sensitivity; a 9% segment move is plausibly material by any reasonable reading.
- Data change: two additional years of training data is a substantive change, not routine.
The team's 'routine retraining' label is not defensible for a methodology change. The honest conclusion: the change was material under the policy's own words, and the absence of a numeric threshold is a policy defect that let the team rationalize.
Note the asymmetry: the team applied a narrow reading (retraining is routine) to a change that was not routine. When a qualitative standard is ambiguous, the conservative reading should govern for regulatory outputs.
2. What should the bank do now, given the five-month window before regulatory review?
Read the worked answer
Act now; don't wait for the review to surface it. Sequence:
- Internal escalation and documentation (immediate): notify model risk management and compliance in writing; open a formal issue with the audit finding. Preserve the changelog and the team's rationale.
- Validate the new model (weeks 1–4): independent validation of the gradient-boosted model, including segment-level output analysis and stability. The 9% segment move needs explanation.
- Assess regulatory impact (weeks 1–3, parallel): determine whether the 4.2%/9% shift affects reported capital ratios and whether the change triggers a notification obligation under the applicable regulatory framework. Compliance owns this determination, not the data science team.
- Decide on disclosure/notification (weeks 3–5): if notification is required, notify proactively with a remediation plan rather than waiting for the review. Proactive disclosure is materially better than being found.
- Remediate the policy (weeks 1–8): add numeric materiality thresholds (e.g., output change beyond X% or methodology class change) and a mandatory routing step so retraining can't bypass model risk management.
- Report to the board risk committee on the gap and remediation.
3. What policy and control changes would prevent recurrence?
Read the worked answer
The failure was a classification and routing gap, so fixes should target how changes are classified and how they reach oversight.
Policy:
- Replace the purely qualitative materiality definition with quantitative triggers plus qualitative ones: e.g., any change in model class, any output change above a stated threshold, any change to training data window beyond N years → material.
- Define 'routine retraining' explicitly and narrowly, with examples of what it does not include.
Controls:
- Change-routing gate: engineering changelogs for models in regulatory scope must route to model risk management automatically; the data science team cannot self-classify a methodology change as routine.
- Segmentation-aware output monitoring: track output shift by segment, not just average, because 4.2% average hid 9% segment moves.
- Pre-change impact assessment: require an estimated output-impact analysis before deploying a retrained model.
- Regulatory calendar integration: tie model changes to the compliance calendar so notification obligations are checked at change time, not at review time.
Assurance:
- Internal audit re-test of the routing control within two quarters.
- A named owner for the materiality determination, with sign-off recorded.