The 98.6% Accuracy That Declined Almost Nobody
A mid-size card issuer has 2.4 million historical applications spanning 30 months, of which 1.35% defaulted within 12 months. The team trains an H2O-3 GBM with balance_classes=TRUE on a random 70/15/15 split with 5-fold cross-validation. Validation AUC is 0.941, and the confusion matrix at the default 0.5 threshold shows 98.6% accuracy. The model goes to the credit strategy team, who run it for a week and report that it declines 0.4% of applicants and is unusable. The risk committee has two hard constraints: the policy must capture at least 60% of eventual defaults, and total declines cannot exceed 8% of applications. Nobody has examined a threshold other than 0.5, nobody has checked whether the random split is appropriate given that months 15–18 were a recession, and nobody has asked what balance_classes did to the predicted probabilities.
1. Explain why 98.6% accuracy is not evidence of a good model, and state what you would report instead. Include the arithmetic that makes the number uninformative.
Read the worked answer
At a 1.35% base rate, a constant 'no default' rule scores 98.65% accuracy - essentially the reported number. The model is therefore indistinguishable from the trivial baseline on the metric the team chose. Accuracy is dominated by prevalence, so it cannot discriminate when the positive class is rare.
What to report instead:
- Recall and precision at the operating threshold, not at 0.5.
- AUCPR, whose no-skill baseline equals the prevalence (0.0135). An AUC of 0.941 says the ranking is good somewhere in the score range; AUCPR says whether it is good where you actually operate.
- KS and lift by decile, which the credit team can translate into a cut-off.
- Cost-weighted expected loss: a missed default costs roughly the outstanding balance, a wrongly declined good customer costs margin plus attrition. The threshold should optimise that, not accuracy.
A useful sanity check before any modelling: compare the trained model's confusion matrix to the all-negative rule at every threshold. If it never beats the baseline by a meaningful margin in the region the business cares about, the metric was the problem, not the model.
2. Given the constraints (≥60% recall, ≤8% decline rate), work out what precision the model must deliver and how you would select the threshold in H2O-3.
Read the worked answer
Arithmetic first. 2.4M applications at 1.35% gives about 32,400 defaults. Capturing 60% means catching about 19,440 of them. Declining at most 8% means at most 192,000 declines. If the model achieves both simultaneously, precision on the declined population is at best 19,440 / 192,000 ≈ 10.1% - roughly ten declines per default caught. Whether that is acceptable is a business question about the value of a saved balance versus the margin lost on a declined good customer; if it is not acceptable, the constraint set is infeasible and must be renegotiated before any modelling work continues.
Threshold selection in H2O-3:
- Compute
h2o.performance(model, valid=TRUE)and read the full threshold table (confusion matrix and metrics at each threshold, e.g.h2o.confusionMatrix(model, thresholds=...)), not the single 0.5 row. - Filter to thresholds where recall ≥ 0.60 and the decline rate ≤ 0.08, then pick the one that maximises precision or minimises expected cost.
- Select on a validation frame that was not used to tune the model, and confirm on the most recent out-of-time period - the recession months shift the base rate and move the operating point.
3. What are the consequences of `balance_classes=TRUE` for the probabilities the business sees, and how do you correct them?
Read the worked answer
balance_classes (or explicit class_sampling_factors) resamples or reweights the training data so the minority class is no longer 1.35% of the rows. The model learns the resampled prior, so its output is no longer an estimate of P(default | features) under the deployment distribution - scores are pushed upward toward the minority class. Two teams can then look at the same model and disagree about what '0.4 probability' means.
Options, in order of preference for a credit use case:
- Train without balancing and move the decision threshold instead. The ranking (AUC, AUCPR) is essentially unchanged, and the probabilities stay interpretable. This is usually the right call when the output feeds a policy cut-off.
- Train with balancing and recalibrate afterwards: fit a monotone mapping (isotonic regression or a one-variable logistic) from out-of-fold predicted scores to observed outcomes on a held-out set that has the true prevalence. Verify with a reliability curve and Brier score.
Never take a threshold learned on balanced scores and apply it to unbalanced production scores, and never present uncalibrated balanced-model probabilities to a regulator or to a downstream expected-loss calculation. Also document which of the two regimes produced the number.