The 94% Dropout Model That Flags Almost Nobody
A district education office in Assam wants to flag students at risk of dropping out before the Class 10 board exams. The dataset covers 4,200 students from 12 schools across three academic years, with 47 columns including attendance percentage, household income band, distance to school, mid-term marks, and appeared_in_final_exam. About 6% of students (roughly 250) actually dropped out.
A two-member team trains a logistic regression and reports 94% accuracy on a random 20% test split. The block officer is impressed and asks for a deployment plan for next term. When you request the confusion matrix, it turns out the model flagged only 30 of the 250 dropouts, and 640 students in total. The team mentions that appeared_in_final_exam and final_marks were kept because they were in the CSV.
The counselling team can realistically follow up with about 300 students per term.
1. The block officer asks why you are not satisfied with 94% accuracy. Using the numbers in the setup, explain what the figure hides and list the metrics that should replace it in the report.
Read the worked answer
Start from the baseline. With 6% prevalence, a model that predicts 'no dropout' for every one of the 4,200 students scores 94% accuracy. The reported figure is therefore indistinguishable from doing nothing, and the officer is being shown a number that the negative class produces on its own.
Then read the confusion matrix as counts. Recall on the dropout class is 30/250 = 12%, so 88% of the students who actually left were missed. Precision is 30/640 ≈ 4.7%, meaning roughly 19 of every 20 students pulled into a counselling queue are false alarms. A high-accuracy model that misses nine in ten positives is useless for the task it was built for.
Replace accuracy with: recall, precision and F1 on the positive class; the confusion matrix in raw counts; PR-AUC rather than ROC-AUC, because ROC curves look flattering at low prevalence; calibration of predicted probabilities; recall at k = 300, since capacity is the real constraint; and subgroup recall by school, gender and income band, because a model can be accurate overall and still fail one school completely.
2. The team proposes dropping the decision threshold from 0.5 to 0.2 so that more dropouts are caught. Evaluate this proposal given that counsellors can follow up with only about 300 students.
Read the worked answer
Lowering the threshold does raise recall, but the arithmetic works against the team. At threshold 0.5 the model already flags 640 students; at 0.2 it will flag a far larger slice of 4,200, plausibly 1,500-2,000. That is five to seven times the counselling capacity. When a list cannot be worked through, the effective recall in the field is not the recall printed in the report; teachers triage by gut feeling, and the model becomes decoration.
Flooding the list also has a cost that the team has not priced. A false positive is a letter home, a home visit, and a teacher who now expects the student to fail. Stigma and self-fulfilling expectations are real harms in a small school.
The defensible design is to treat the output as a ranking, not a classification. Score every student, sort descending, and send the top 300 each term. Report recall@300 (for example, 130 of 250 dropouts inside the top 300 = 52%) and let the threshold be derived from capacity rather than chosen by convention. A second, informal watchlist can absorb scores just below the cut-off without triggering formal labels.
3. Identify every problem with how the model was built and evaluated, and describe the corrected experiment you would run before any deployment.
Read the worked answer
Two problems sit in the features. appeared_in_final_exam and final_marks are recorded after the dropout event, so they encode the label directly. Any model using them will look excellent and be worthless at prediction time. The rule is to keep only what is knowable at the moment of the flag, for example data up to July of Class 10, and to delete every column created after that date.
The split is also wrong. A random 80/20 split puts students from the same school and the same year on both sides, so the model memorises school-specific patterns and the test score is optimistic. Use a time-based split: train on the two earlier years, test on the most recent one, and group by school so no school appears in both sets. Tune hyperparameters and choose the threshold on a validation slice inside the training years, never on the test year.
Finally, report per-school metrics, because deployment is per school, and re-check prevalence drift each year. The headline claim should be something like 'recall of 52% at 300 flags on the 2023-24 cohort, up from the 12% baseline', not '94% accurate'.