The Explainability Request That Exposes the Model's Weakness
A regional bank uses a gradient-boosted model to approve or decline small-business credit lines up to $150,000. A denied applicant, a bakery owned by a recent immigrant, requests an explanation. The bank's compliance team runs a SHAP analysis and finds the top negative contributors are 'years in business' (2.1 years) and 'industry code - food service.' The model's overall approval rate is 64%, but for food-service businesses with under three years of operation it is 31%. The bank's own policy says it does not lend to businesses with under two years of operation, but this applicant is above that floor. The data science lead notes that SHAP explains the model, not the decision, and that the 'industry code' feature is a coarse bucket that lumps together very different businesses. The bank must respond within the regulatory window.
1. Distinguish between explaining the model and explaining the decision, and say which the applicant is owed.
Read the worked answer
Explaining the model means describing global behavior: which features matter across the population, how the model weights them, what the overall approval rate is. SHAP values are a model-level artifact - they tell you the model's internal attribution, which can be unstable, correlated, and sensitive to feature encoding.
Explaining the decision means giving this applicant the specific, actionable reasons their application was declined, in terms they can act on or contest. That is what adverse-action rules contemplate, and it is what a person can use.
The applicant is owed a decision explanation, not a model explanation. Concretely: 'Your application was declined primarily because the business has 2.1 years of operating history, below the profile the model associates with repayment, and because food-service businesses in your segment have shown higher default rates in our portfolio.' That is actionable - the applicant can add a guarantor, wait, or provide supplemental financials. 'SHAP value of -0.34 on feature 17' is not.
The data science lead is right that SHAP is not the decision, but that is a reason to translate, not a reason to withhold. The bank should map the top contributors to plain-language reasons and confirm they match the actual policy grounds.
2. The 31% approval rate for young food-service businesses is well below the 64% overall. Is this evidence of discrimination, and what should the bank do?
Read the worked answer
It is evidence of disparate impact that requires justification, not proof of intentional discrimination. The distinction matters for what the bank does next.
- Is the feature job-related? 'Years in business' is a classic, defensible credit factor - young firms do fail more. If the bank can show the 31% rate reflects genuine default risk in that segment, the factor is defensible.
- Is 'industry code - food service' too coarse? A bucket that lumps a bakery, a catering operation, and a food truck is not a meaningful risk category. If the code is doing work that finer, job-related variables could do better, the coarse bucket is a weak justification.
- Is there a proxy problem? The applicant is a recent immigrant. If 'years in business' or 'industry code' correlates with national origin in a way that is not explained by credit risk, the bank needs to test that directly - compare approval rates and default rates by national origin where data is lawfully held.
What to do: (1) produce the segment-level approval and default data to test whether 31% tracks risk; (2) replace the coarse industry bucket with finer, justified categories; (3) run a fair-lending disparate-impact review with counsel; (4) document the business necessity for each factor. If the 31% cannot be tied to risk, the bank has a problem regardless of intent.