Feature leakage and reproducibility in a claims severity model
An actuarial data science team is building a feature pipeline for a household claims severity model. The repository contains 14 Jupyter notebooks, 320 engineered features and no versioned training data. The best model reports validation AUC 0.91, but when re-run by a colleague the AUC is 0.76. Investigation shows that a 'days_since_last_claim' feature was calculated using the claim date, and several features were fitted on the full dataset before the train-test split. The team also overwrites the production feature table weekly, so backtests cannot be reproduced exactly. The model is due to go live in six weeks and will feed reserve estimates. The chief actuary requires reproducibility, auditability and clear ownership. Two analysts have left and their notebooks are poorly documented. A previous model failed validation for similar reasons.
1. Diagnose the likely causes of the performance gap and propose a corrected evaluation approach.
Read the worked answer
The 0.91 AUC is probably leakage, not skill. 'days_since_last_claim' calculated using the claim date incorporates information only known after the claim, so it leaks the target. Fitting features on the full dataset before splitting lets validation-fold statistics influence training. Re-running in a different notebook state changes preprocessing and random seeds, explaining 0.76. Correct by defining a prediction-time data cut-off, excluding post-event fields, fitting all imputations and encodings inside cross-validation, and splitting by time or policyholder rather than randomly. Rebuild the target and check for duplicate claims across folds. Report the corrected AUC and calibration with confidence intervals. The go-live decision should be reset: reserve estimates need a stable, auditable model, not a notebook score.
2. Design the data and feature governance controls needed before go-live.
Read the worked answer
Put code in Git with peer review; version data and feature definitions in a feature registry; use a pipeline that is deterministic from raw data to score. One named owner for the feature table and one for the model. Add automated tests for leakage, schema, ranges and row counts; lock environments; store model artefacts and training data snapshots; log every production score with model and data version. Backtests must run from the same snapshot. Separate development, validation and production permissions. Document lineage for each feature, including when it becomes available. Before go-live, require a reproducibility test: an independent analyst must recreate the validation result from the repository.
3. What should be reported to the chief actuary, and what recommendation would you make about the six-week go-live date?
Read the worked answer
Tell the chief actuary that the current validation overstates performance, the model is not ready to feed reserves, and the six-week date should be treated as at risk. Present the corrected validation, the reserve impact of using the model versus the existing process, and the residual model risk. Recommend a remediation plan with sign-off criteria: leakage-free pipeline, independent reproduction, documented ownership, backtest on historical snapshots, and a fallback to the existing method. If reserves must be produced before remediation, use the existing method or a conservative adjustment, and disclose the model limitation. Do not let the reported 0.91 AUC enter the model risk report without correction.