A training score hides the evaluation boundary
A fictional team reports a strong model score after selecting features and preprocessing values using the complete dataset, including records later labeled as the test set. The final evaluation is described as independent because the model was not directly fitted to those test labels in its last training call. A colleague asks whether earlier decisions had already used information from the purported holdout. The supplied record does not establish a valid fresh evaluation. This written exercise trains no model and makes no decision about people. It asks the learner to identify leakage beyond the final fitting call and explain how an appropriate evaluation design should preserve the intended information boundary.
1. Why is the independence claim unsupported?
Read the worked answer
Preprocessing or feature selection using the complete dataset can leak information from the supposed holdout. Independence is not established solely by the last fit call’s input labels.
2. What would a suitable review require?
Read the worked answer
Fit relevant transformations and selection only within the permitted training/validation procedure, respect related/time-ordered records and evaluate with a properly protected final set and task-appropriate measures.