Reason Codes Under Audit
A lender uses Driverless AI to build a model that recommends credit-limit decreases. Compliance requires that every declined account receive the top four reason codes in plain language, and that those reasons be 'the factors that most increased the likelihood of the decline'. The production model is a DAI ensemble of XGBoost and TensorFlow models with AUC 0.884. credit_utilization and num_late_payments_6m correlate at 0.71; income and zip_median_income correlate at 0.83. The MLI tab shows a partial dependence plot for num_late_payments_6m that is flat below 2 and then jumps sharply, while the SHAP summary ranks the same variable as the top global driver. Reason codes are generated from DAI's K-LIME surrogate with four bins. An internal audit samples 500 declined accounts and finds 61 cases (12.2%) where the top reason contradicts the account's most recent adverse action letter, and 18 cases where the reason given is a variable the customer cannot influence (zip_median_income).
1. Explain the difference between K-LIME reason codes and SHAP attributions. Why might they disagree, and which should drive an adverse action notice?
Read the worked answer
K-LIME fits a linear surrogate weighted to a neighbourhood of the record and reads the coefficients as reasons. The output explains the surrogate, not the ensemble: if the local linear fit is poor - very likely for a 300-tree XGBoost plus a neural net with four bins - the reasons describe a model that was never deployed. It is fast and stable, which is why it is the default, but its fidelity to the actual score varies per record and is usually not reported alongside the codes.
SHAP contributions (for tree models, exact TreeSHAP via h2o.predict_contributions or DAI's Shapley option) attribute the prediction additively: contributions sum to the score minus the baseline. That reconciliation property is exactly what a compliance function wants, because the reasons can be shown to add up to the decision.
For adverse action, prefer the additive, exact attribution, and where a surrogate is used, report its local fidelity (R² against the true score in the neighbourhood) per record and suppress or escalate reason codes where fidelity is low. The 12.2% contradiction rate is consistent with surrogate misfit, not necessarily with a modelling error - but either way the notices are being justified by something other than the score.
2. What explains the behaviour of `income` and `zip_median_income` in the attributions, and how do you make the reason codes actionable and defensible?
Read the worked answer
When two features carry nearly the same information, SHAP splits the credit between them in a way that depends on the correlation structure and the background dataset. Neither receives 'the' true share, because with correlated inputs the attribution is not unique - the model can substitute one for the other without changing the prediction. The practical consequence is that the top-four list can be filled by two spellings of the same concept, crowding out genuinely distinct reasons.
Remedies:
- Aggregate by concept before ranking. Group correlated features (income + zip median income; utilisation + late payments) and sum their contributions, then take the top four concepts and express each in plain language.
- Remove non-actionable and proxy variables from the reason-code generation.
zip_median_incomeis both non-actionable and a proxy for protected characteristics, which is a fair-lending exposure regardless of its predictive value. It can stay in the model if the model is validated for disparate impact, but it should not appear in a customer-facing notice. - Tie the reason text to a defined counterfactual - 'utilisation is 78%, above the 65% threshold used in the decision' is defensible; 'zip median income contributed +0.31' is not.
Finally, run DAI's disparate impact analysis on the retained feature set and keep the results with the model documentation.
3. Why might the partial dependence plot for `num_late_payments_6m` be flat and then jump, and how would ICE plots change your interpretation?
Read the worked answer
PDP sets the feature to each value for every row and averages the predictions, which marginalises over everything else. Two things produce the flat-then-jump shape. First, interactions: if late payments only matter when utilisation is high, averaging across low-utilisation accounts dilutes the effect at low values and the curve collapses toward the mean. Second, sparse support: if few accounts have more than four late payments, the right-hand part of the curve is computed on a handful of rows or pure extrapolation and should not be read as a model behaviour at all.
ICE plots show one curve per record instead of the average. They distinguish three cases that the PDP cannot: a genuine threshold that nearly everyone shares (curves stacked and parallel), heterogeneity where the average is meaningless (curves crossing or fanning), and a region where almost no records exist (few or no curves).
Practical resolution: plot ICE with a centred PDP overlay, restrict the x-axis to the observed support and annotate the count of records at each value; if the curves are heterogeneous, report a segmented PDP (e.g. by utilisation band) rather than a single global curve, and do not use the global shape to justify a policy threshold. This matters directly here, because a reason code derived from a spurious PDP feature shape will not survive scrutiny.