Nine Exceptions, All in One Bad Week
A $800m multi-asset portfolio runs a 99% one-day VaR from 60-day equally weighted historical simulation, with a 12% annualised volatility target. Over the last 250 trading days there were 9 VaR exceptions against an expectation of 2.5. Six of the nine fell inside a single eight-day window four months ago, during a sharp simultaneous move in rates and equities; the other three were isolated. Four months ago the portfolio also added a 15% allocation to EM local-currency debt, and the historical window still contains several months of unusually calm markets from before that change. The risk committee asks whether the model should be replaced, recalibrated, or retained with a larger limit buffer, and the CRO wants the answer in terms of both statistical evidence and the ten-day liquidation horizon used in the liquidity policy.
1. Compute the Kupiec proportion-of-failures statistic and interpret it. How much power does this test actually have at n = 250?
Read the worked answer
Kupiec POF likelihood ratio. With x = 9, n = 250, p = 0.01:
LR = 2[x·ln(x/np) + (n−x)·ln((n−x)/(n(1−p)))] = 2[9·ln(9/2.5) + 241·ln(241/247.5)] = 2[9(1.2809) + 241(−0.026616)] = 2[11.528 − 6.414] = 10.23
Against χ²(1) at 1% = 6.635, this rejects the null of correct unconditional coverage decisively (p ≈ 0.0014).
Power. The test is notoriously weak at n = 250. Evaluating the same statistic: 7 exceptions gives LR ≈ 5.50 (fails to reject at 5%); 8 exceptions gives LR ≈ 7.74 (rejects). So the test only begins to fire at 8 exceptions, and 250 days is roughly the minimum sample anyone should backtest on. The practical reading is the reverse of the usual one: because the test is blunt, 9 exceptions is a strong signal, not a marginal one. The model is not defensible as calibrated.
Caveat. POF tests only the unconditional rate. It is silent on when the failures occurred, which is the more damaging fact in this case.
2. What does the clustering of exceptions imply that the POF test misses, and why does it matter more for a ten-day liquidation horizon than for a one-day number?
Read the worked answer
Clustering means losses are not iid. Six of nine breaches in an eight-day window is the signature of a volatility regime, not of a well-calibrated model with bad luck. The appropriate test is Christoffersen's conditional coverage (independence of exceptions), which would reject far more strongly than Kupiec here. The economic content: the model is roughly right on average across a year and badly wrong exactly when it matters.
Why the horizon changes the answer. A one-day 99% VaR assumes you can de-risk overnight. The liquidity policy uses a ten-day horizon, and under a regime where daily moves are serially correlated, ten-day losses scale faster than √10. Concretely: if the model's 1-day VaR was, say, 0.9% of NAV, the √10 scaling gives 2.85%; but the days inside a crisis are correlated, so the realised ten-day loss in that window behaves more like 10 × the daily move than √10 ×. The gap between a 2.85% ten-day estimate and a clustered realisation of 6–8% is the real capital risk.
Second-order point. The 60-day equally weighted window decays slowly (1/60 per day) and, following a calm period, still carries stale low-volatility observations four months after the EM debt allocation was added - a new risk factor that the historical window barely contains. The model is simultaneously too slow to react to a regime and missing a new exposure.
Implication. The exception count understates the problem. Reporting only the 9-vs-2.5 comparison to the board would be a governance failure.
3. Recommend a concrete course of action. State what you would change, what you would keep, and what evidence would convince you the fix worked.
Read the worked answer
Change:
- Replace equally weighted historical simulation with filtered historical simulation: EWMA volatility (λ ≈ 0.94) applied to standardised historical residuals, so current volatility, not the 60-day average, drives the scale. This directly addresses the clustering failure.
- Re-estimate the correlation/co-movement structure on a shorter, regime-aware basis and explicitly include the EM local-currency sleeve - either with its own history or by stress-mapping from a proxy. A factor that has been in the book for four months cannot be estimated from a window that does not contain it.
- Replace the single 1-day number with an expected shortfall and a 10-day liquidity-horizon measure consistent with the policy, and add a stress scenario that replays the eight-day window.
- Apply a conditional-coverage overlay: a traffic-light or a scaled buffer that widens when exceptions cluster, not merely when the annual count is high.
Keep: the 99% confidence level and the historical-simulation backbone (residual-based simulation preserves the empirical fat tails better than a normal assumption).
Evidence the fix worked:
- Rolling out-of-sample backtest of the new model over the same 250 days and the prior three years, with Kupiec and Christoffersen tests reported together.
- Confirmation that the six clustered exceptions are no longer exceptions, without the model now generating excessive exceptions in calm markets (a fix that just doubles every VaR is not a fix).
- A 12-month monitoring commitment with a stated escalation trigger - e.g. more than 2 exceptions in any rolling 20-day window triggers immediate review regardless of the annual count.
Finally, until the new model has run in parallel for at least one quarter, keep the existing limit and apply a discretionary buffer sized to the observed understatement rather than the average.