Class Imbalance and Threshold Selection Under a Cost Constraint
A team builds a binary classifier for equipment failure on 2.1 million sensor windows, of which 0.9% are failures. They train XGBoost with default settings and get ROC-AUC 0.94 but recall 0.31 at the default 0.5 threshold. The business cost of a missed failure is roughly $8,000 in downtime; a false alarm triggers a $300 inspection. The team proposes oversampling the minority class 10x and reporting accuracy, which currently sits at 99.1%. A holdout is split randomly by window, but windows from the same machine are highly correlated and some machines appear in both train and holdout.
1. Critique the proposed evaluation plan and specify a better split and metrics.
Read the worked answer
Accuracy at 99.1% is uninformative: predicting 'never fails' scores 99.1%. Oversampling 10x changes the base rate the model sees, so predicted probabilities no longer reflect reality unless recalibrated, and it inflates apparent recall if evaluation is done on the resampled distribution. Reporting accuracy after oversampling compounds both problems.
The split is the more serious flaw. Windows from the same machine are correlated, so a random window split leaks machine-specific signatures into the holdout and makes performance look better than it will be on unseen machines. Split by machine (grouped split), and if deployment targets new machines, hold out entire machines. If deployment targets the same fleet over future time, use a time-based split.
Metrics: report precision, recall, and PR-AUC rather than ROC-AUC alone, since ROC-AUC is optimistic under heavy imbalance. Because costs are asymmetric, convert the model output to expected cost: choose the threshold that minimizes 8,000 * false negatives + 300 * false positives on the holdout, and report the resulting confusion matrix, expected cost per 1,000 windows, and calibration (reliability curve) so the threshold is meaningful. Keep a no-oversampling baseline for comparison.
2. Given the costs, what threshold policy would you recommend, and how would you validate that it generalizes?
Read the worked answer
The cost-optimal decision rule is to predict failure when p * 8,000 > (1 - p) * 300, i.e., p > 300 / 8,300 ≈ 0.036. So the threshold should be near 0.036, far below 0.5 - which explains why recall is 0.31 at 0.5. Rather than hard-coding from theory, sweep thresholds on the holdout and pick the one minimizing total expected cost, then confirm it lands near the analytic value; a large divergence signals miscalibration.
Calibration matters: if the model was trained with 10x oversampling, its probabilities are shifted and the analytic threshold will not apply. Either recalibrate (isotonic or Platt on a held-out set) or compute the threshold empirically on the untouched holdout, not the resampled one.
Validate generalization with grouped cross-validation by machine and a time-based final holdout. Report the cost curve (expected cost vs. threshold) and its variance across folds; if the optimal threshold moves a lot between folds, prefer a slightly more conservative threshold and monitor. In production, track the predicted-positive rate and the realized false-alarm and miss counts, and re-derive the threshold when the failure base rate or inspection cost changes.