Predicting Loan Default with Imbalanced Data
A financial institution wants to predict loan default using historical data of 50,000 loans, of which only 2% defaulted. They plan to use Alteryx Machine Learning to build a classification model. The business goal is to identify high-risk loans to manually review before approval. The cost of a false negative (approving a loan that defaults) is estimated at $10,000, while a false positive (rejecting a good loan) costs $500 in lost opportunity. The team is considering using accuracy as the primary evaluation metric because it is easy to explain to stakeholders.
1. Evaluate the appropriateness of using accuracy as the primary metric for this problem. What alternative metric(s) would you recommend and why?
Read the worked answer
Accuracy is inappropriate because with 2% default rate, a naive model predicting 'no default' for all loans would achieve 98% accuracy but would fail to identify any defaulters, which is the entire purpose. The costs of errors are highly asymmetric: a false negative costs $10,000, while a false positive costs $500. Therefore, we need a metric that focuses on the positive class (default). Recall (sensitivity) is crucial because we want to catch as many defaulters as possible. However, precision is also relevant because false positives have a cost. A better approach is to use a cost-sensitive metric or to optimize for a specific recall level while monitoring precision. Alternatively, we can use the F-beta score with beta > 1 to emphasize recall, or use expected cost as a custom metric. In Alteryx Machine Learning, we can set the positive class and evaluate using precision-recall curves, and choose a threshold that balances the costs. The team should also consider using SMOTE or class weights to address imbalance.
2. How would you handle the class imbalance during data preparation and model building in Alteryx Machine Learning? Describe specific steps.
Read the worked answer
In Alteryx Machine Learning, we can address class imbalance by: (1) Using the 'Sample' tool to perform stratified sampling to ensure both classes are represented in training and validation sets. (2) Applying oversampling of the minority class (e.g., SMOTE) using the 'SMOTE' tool or the 'Oversample' tool in the Prepare phase. (3) Alternatively, using class weights in the model configuration if supported (e.g., in XGBoost, set scale_pos_weight). (4) Evaluating models using precision-recall curves and AUC-PR rather than ROC-AUC, which can be misleading with imbalanced data. (5) Adjusting the classification threshold based on the cost matrix. (6) Using cross-validation with stratification to ensure reliable estimates. It's also important to reserve a holdout set with the original distribution for final evaluation.