The 92% Model That Catches Almost No Failures
A coaching centre in Siliguri has three years of records for 4,800 students who sat its final mock test. Only 384 (8%) failed. A teacher builds a logistic regression model in scikit-learn, splits the data randomly 80/20, and reports 92% accuracy. The director is delighted and wants to use the model to send extra remedial classes to students predicted to fail. A second teacher notices that one input feature, 'mock_test_score', is recorded after the final mock, and another, 'remedial_class_attended', is filled in only for students who were already flagged by teachers during the year. The model uses both. Meanwhile, in the 20% test set the model flags 41 students as failing, of whom 9 actually failed; 77 students in that test set failed in total. Nobody has computed any metric other than accuracy. The dataset has no missing values, which the teacher cites as further proof of quality.
1. The director says the 92% accuracy proves the model works. Using the confusion-matrix figures in the setup, explain what accuracy is hiding and state the metrics you would report instead, with values.
Read the worked answer
The fail class is only 8% of the data, so a model that predicts 'pass' for every single student already scores about 92%. Accuracy is being measured against a 92% majority-class baseline: the model has learned to say the common thing, not to make the decision the centre actually cares about.
Reconstruct the test-set confusion matrix (960 students, 77 real failures):
- flagged fail and did fail (TP) = 9
- flagged fail but passed (FP) = 41 − 9 = 32
- failed but not flagged (FN) = 77 − 9 = 68
- passed and not flagged (TN) = 960 − 41 − 68 = 851
Accuracy = (851 + 9) / 960 = 89.6% - actually slightly worse than the baseline. Precision(fail) = 9/41 = 22%: nearly four out of five students sent to remedial classes would not have needed them. Recall(fail) = 9/77 = 11.7%: the model misses about 88% of the students it exists to help. F1 ≈ 15%.
Report the full confusion matrix, precision/recall/F1 for the fail class, PR-AUC (ROC-AUC is flattering under heavy imbalance), and the threshold used. Every headline number should be printed next to the majority-class baseline, otherwise the claim is meaningless.
2. Identify the data leakage in the feature set and describe how you would rebuild the training and test sets so the reported score is trustworthy.
Read the worked answer
Two columns are not available at the moment the prediction has to be made.
- mock_test_score is recorded after the final mock test, i.e. after the outcome window the model is supposed to forecast. Including it is close to reading the answer sheet.
- remedial_class_attended is generated by teachers' earlier informal flags, so it encodes a human judgement that itself depends on how the student eventually performed. It is a proxy for the label, and it will not exist in the right form for next year's students anyway.
The claim that 'no missing values means good data' is also false - complete columns can still be contaminated or sentinel-filled.
Rebuild as follows. Fix a prediction point (for example, the end of Term 1). Rebuild the feature table using only information that exists before that point, and confirm availability with the teachers column by column. Drop the two leaking columns, or replace them with genuinely prior signals (attendance in Term 1, marks of the previous annual exam, distance from home). Use a time-based split - train on 2021-22 and 2022-23, test on 2023-24 - instead of a random split, and keep the fail class stratified. Expect the accuracy to fall sharply; that fall is the honest estimate, not a regression.
3. If the centre still wants to run the model next term, what deployment safeguards would you insist on, given the relative costs of the two error types?
Read the worked answer
Start from the cost asymmetry. A false negative (a student who will fail, not flagged) costs a lost chance at remedial help; a false positive (a student flagged unnecessarily) costs a few extra classes and some stigma if handled clumsily. Under that asymmetry the default 0.5 threshold is wrong - it should be lowered until recall(fail) is acceptable, and the resulting precision reported honestly so staff know how many flagged students are false alarms.
Safeguards worth insisting on:
- Human decision, not automatic action. The score should trigger a teacher conversation, never an automatic placement or a label attached to the student.
- Explain what is available. Show the two or three strongest contributing factors per student so a teacher can sanity-check; a bare 0-100 score invites automation bias.
- No permanent record. Scores should expire each term and must never be shown to parents or other students as a verdict.
- Monitoring. Log precision/recall against actual results each term, disaggregate by class and section, and re-check for drift when the syllabus or intake changes.
- Appeal route. A student or guardian must be able to ask for the flag to be reviewed by a person.
- Retrain discipline. Re-run the leakage audit before every retraining cycle; leakage creeps back in whenever new columns are added.