Calculators or Models? Scoping 31 Unregistered ML Tools at an Asset Manager
Ardent Asset Management ($250 billion AUM) has a model risk policy that applies to 'models used to measure risk, value positions, or calculate regulatory capital'. Its nine investment teams - which collectively beat the benchmark by 40 basis points last year - have built 31 machine-learning signals and tools outside that scope, including a retrieval-augmented research assistant used by 40 analysts. Last month the assistant summarised a broker note and produced an issuer revenue figure that was not in the source; the figure reached a portfolio manager's pre-trade memo before an analyst caught it. Internal audit issued a finding that AI use is 'pervasive, undocumented, and unvalidated'. The CIO argues that requiring independent validation of research tools would slow delivery and that 'these are calculators, not models'. The CRO wants all 31 in scope; the model risk team has four full-time validators and a 14-week backlog. The board risk committee meets in nine weeks and has asked for a position paper.
1. Is the CIO's 'calculators, not models' argument defensible? Identify the root cause and set out the scope definition you would recommend.
Read the worked answer
Not defensible on the facts, though it contains a legitimate concern about proportionality that should be answered rather than dismissed.
Why the argument fails The research assistant produced a factual assertion that reached a pre-trade memo. Influencing an investment decision with an unverified output is precisely the harm model risk management exists to control - the label 'calculator' does not change the causal chain. The old scope boundary (risk measurement and regulatory capital) was drawn for prudential models; it was never a finding that investment tools carry no risk.
Root cause is definitional and procedural, not behavioural
- The policy defines a model by technique and purpose, so ML tools with real failure modes fall outside.
- There is no intake route for bottom-up builds and no approval gate for procuring AI SaaS.
- There is no channel for an analyst to report a bad output, which is why the error surfaced by luck rather than by design.
Recommended scope Define in-scope as any algorithmic system - vendor, embedded, generative or home-grown - that materially informs an investment, client, risk or operational decision. Scope by decision impact, not technique. Add a days-long self-classification intake, and extend the acceptable-use policy to cover the research assistant case.
Test of judgement: do not scope by the 'AI' label. A spreadsheet heuristic embedded in position sizing can matter more than a neural net; a definition that captures only 'AI' leaves the next gap open.
2. Given four validators and a 14-week backlog, design a proportionate tiering and validation model. What do you do about the research assistant this week?
Read the worked answer
Blanket independent validation of 31 tools would be theatre, not assurance - the backlog guarantees the risky ones wait longest.
Tiering by decision impact
- Tier 1 (independent validation before use): directly generates trade signals or client-facing output. Expect roughly six to eight of the 31 on triage.
- Tier 2 (decision support with human review): documented testing and peer review by the team, sampled and challenged by model risk.
- Tier 3 (productivity): inventory entry, acceptable-use attestation, spot checks.
Resourcing Sequence the tier 1 set by exposure, not by which team shouts loudest. Use co-sourcing or contract validators for surge, and keep validation independent of the builders - a validator who reports to the investment team is not a validator. Publish minimum testing standards per tier so a 14-week waterfall is not the only path to sign-off.
This week on the research assistant Require source citation with every numeric output and verified retrieval, or disable numeric generation entirely; log outputs; brief the 40 analysts. Then run a tier 2 review with a 100–200 output sample testing retrieval quality and citation fidelity - hallucinated attribution is the known failure mode of retrieval-augmented systems, and it is testable.
Cost of delay, stated honestly If a Tier 1 tool goes live before validation, record the residual risk and have the accountable executive sign it. Undocumented tolerance is the actual control failure; documented tolerance is a decision.
3. What should the position paper to the board risk committee contain, and how would you handle the CRO/CIO disagreement?
Read the worked answer
Reframe it: this is not a personality conflict, it is one risk question with a proportionate answer. The CIO's speed concern is real and can be met by tiering; the CRO's scope concern is also real and cannot be met by a 14-week queue.
Paper contents
- Evidence: the hallucinated figure and how it reached a pre-trade memo, the audit finding, the count of unregistered tools, and a stated caveat that 31 is likely not the true total.
- The gap: policy definition and the absence of an intake and procurement gate - presented as a system defect, not a team failure.
- The proposal: revised scope, tiering model, effective date, and what changes for each team.
- Resourcing truth: four validators cannot cover 31 tools. Present options - extend the timeline, fund co-sourcing, or accept documented residual risk in writing - with a recommendation, not a menu.
- 90-day plan: triage, tier 1 validation, policy amendment, training - with named owners and dates.
- Decision requested: approve the scope change and interim standard; note the residual-risk position for tools running unvalidated.
Handling the disagreement Surface the independence dimension: models owned by the business that reports P&L require second-line challenge free of that reporting line. Put a name against each residual risk - boards act on named accountability, not on committee consensus. Avoid presenting the CRO as the brake and the CIO as the accelerator; both positions will look reasonable if the paper is written around tiers rather than around people.