Selecting and explaining a more flexible model
A team compares a small tree, a random forest, and a gradient-boosted model for the same defined insurance target. The deepest tree nearly eliminates training error but performs poorly on a later validation period. A colleague claims that adding enough trees must eliminate all uncertainty, even though the fitted model errors are strongly correlated. The boosted model is tuned repeatedly against a sample originally labelled final test. Its feature-importance chart is then compared numerically with a different importance method from the forest. Two vehicle attributes are strongly associated, and a partial-dependence chart includes combinations rarely observed together. Management wants a model choice and an explanation that are meaningful for deployment rather than simply the most impressive training fit.
1. What modeling and validation changes are needed?
Read the worked answer
Use appropriate held-out validation to control complexity, stage count, and learning rate, and restore a genuinely untouched final evaluation or an appropriate outer procedure after repeated tuning. Correlated errors limit averaging benefits, so ensemble size alone does not guarantee certainty or future accuracy.
2. How should the explanation be qualified?
Read the worked answer
State the importance definition and evaluation sample instead of comparing raw values from different methods as identical quantities. Examine correlated-feature substitution and unrealistic partial-dependence combinations. Describe explanations as properties of the fitted prediction system, not proof that changing a feature causes the outcome.