MLARM common mistakes
MLARM · Understand the traps and how to avoid them.
Treating model risk management as a pure model-performance exercise, so when a scenario describes a well-performing model used for a high-impact decision, the candidate concludes that no model risk exists and skips the governance, documentation, and validation controls.
Most study material emphasizes accuracy metrics and algorithm families, so candidates mentally equate 'good AUC' with 'low risk' and forget that model risk is defined by the potential for adverse consequences from decisions or estimates, independent of how well the model scores.
What to do instead
Before answering any scenario, apply a fixed two-axis test: (1) what decision or estimate does this model drive, and how material is the harm if it is wrong; (2) what is the model's inherent complexity and data dependency. Risk tier flows from both axes. Practice by writing a one-line risk tier and justification for every practice scenario before looking at the options.
Confusing the roles of the three lines of defense: describing the first line as independent validation, assigning model ownership to internal audit, or placing validation inside the business unit that built the model.
The terminology (development, validation, audit) sounds interchangeable to candidates who have not internalized that independence is the defining property. Options that use the right words in the wrong organizational slot look plausible.
What to do instead
Memorize a rigid mapping: first line = model owners/developers who build and use the model and own its documentation; second line = independent model risk management/validation that challenges but does not build; third line = internal audit that assesses the process itself. For any option, ask 'is this actor independent of the model's development and use?' If not, it cannot be validation.
Treating regulatory frameworks as interchangeable checklists, so a question asking which framework applies to a given institution or risk type gets answered with whichever framework name the candidate remembers best (for example, citing a banking model risk framework for an insurer or a voluntary ethics guideline as if it were binding law).
Candidates memorize framework names and headline principles but not their scope, issuer, and legal status, so they pattern-match on keywords like 'AI risk' or 'governance' instead of jurisdiction and sector.
What to do instead
Build a one-page framework map with four columns: issuer, sector/jurisdiction scope, binding vs. voluntary, and core obligation. Before answering, identify the institution type and jurisdiction in the stem, then filter frameworks by scope first and content second. Distinguish supervisory guidance, statutes, and voluntary standards explicitly in your notes.
Assuming that removing direct identifiers (names, account numbers) makes a dataset de-identified and therefore outside privacy controls, ignoring quasi-identifiers, linkage attacks, and re-identification risk in the modeling data.
Candidates conflate 'no direct identifiers' with 'anonymous'. Privacy training often focuses on direct identifiers, and model-development scenarios emphasize feature engineering rather than privacy classification.
What to do instead
Use a two-step test on any data scenario: does the dataset contain direct identifiers, and separately, can combinations of remaining fields (age, ZIP, date, transaction patterns) single out individuals or be joined to external data? If re-identification is plausible, treat the data as personal data requiring purpose limitation, minimization, and access controls even after masking.
Answering data-quality questions by naming a metric (completeness, accuracy) without connecting it to the specific modeling use, such as defending a high missing-data rate because imputation exists, or approving training data with label leakage or survivorship bias because it is 'large and clean'.
Candidates learn data-quality dimensions as a vocabulary list and apply the most familiar one. Volume and surface cleanliness feel like quality, and imputation feels like a solved problem.
What to do instead
For each data scenario, ask three questions in order: is the data representative of the population the model will score; are the labels correct and free of leakage from post-outcome information; is missingness random or informative? Write the answer to each before evaluating options, and reject options that treat imputation or dataset size as a substitute for representativeness and label integrity.
Treating validation as a one-time pre-deployment gate, so an option that says 'the model was validated at approval, therefore no further validation is needed' is selected, and ongoing monitoring, outcomes analysis, and revalidation triggers are overlooked.
The word 'validation' evokes a test or exam that happens once. Candidates underweight lifecycle language and forget that material changes, drift, or new use cases reset the validation obligation.
What to do instead
Anchor on the principle that validation is a continuous obligation with defined triggers. Memorize a trigger list: material change in data, population, or use; performance degradation beyond tolerance; drift alarms; new regulatory expectations; periodic scheduled review. When an option claims a past validation suffices, look for the missing trigger analysis and reject it.
Reading drift as equivalent to performance decay, so a scenario showing stable accuracy but shifted input distributions is judged 'no issue', or a scenario with degraded accuracy but stable inputs is attributed to drift rather than to label or concept change.
Monitoring dashboards commonly display both drift and performance, and candidates collapse them into one concept. The distinctions among data drift, concept drift, and label drift are subtle and rarely practiced.
What to do instead
Separate the signals explicitly: input/data drift = change in feature distribution; concept drift = change in the relationship between features and target; label drift = change in the outcome distribution; performance decay = change in measured accuracy against outcomes. In each scenario, name which signal the evidence shows, then choose the response that matches that signal rather than a generic 'retrain'.
Treating explainability as a compliance checkbox and selecting the most sophisticated technique (for example, a post-hoc local attribution method) regardless of the audience, the model type, or whether the explanation is faithful to the model's actual behavior.
Candidates associate explainability with tool names and assume more advanced equals better, without considering that a regulator, a customer, and a data scientist need different explanations, and that some post-hoc methods can misrepresent intrinsically interpretable models.
What to do instead
Answer explainability questions by first naming the audience and the decision at stake, then matching the method: intrinsically interpretable models where stakes are high and transparency is required, global feature importance for model oversight, local attributions for individual adverse-action explanations, and documented limitations for any post-hoc method. Reject options that claim a post-hoc explanation is a complete substitute for model transparency.
Handling fairness by choosing a single statistical parity-style metric and declaring the model fair, or by assuming that removing protected attributes from the feature set eliminates bias.
Candidates memorize one or two fairness metrics and treat them as universal, and the intuition that 'not using race or gender' prevents discrimination is strong even though proxies and proxy variables preserve disparate impact.
What to do instead
For any fairness scenario, first identify the legal or policy standard in play (disparate treatment vs. disparate impact, equal opportunity, calibration by group), then check whether the chosen metric actually matches that standard. Explicitly test the proxy hypothesis: excluded attributes can be reconstructed from correlated features. Select options that measure outcomes across groups and document trade-offs among metrics rather than asserting a single metric proves fairness.
Underweighting third-party and vendor risk: accepting a vendor's own validation results as independent validation, or failing to require contractual rights to model documentation, audit access, and change notification for a purchased or outsourced model.
Vendor materials arrive polished and pre-validated, and candidates assume the vendor's testing transfers to the institution. The governance gap-who owns the model risk when a third party builds it-is easy to miss in scenario wording.
What to do instead
Apply the rule that outsourcing the model does not outsource the risk: the institution remains accountable. For vendor scenarios, check four things-independent validation or effective challenge by the institution, documentation and transparency rights, audit and inspection rights, and notification of material changes or version updates. Reject any option that treats vendor-provided evidence as a substitute for the institution's own challenge.