The Capstone That Scored 0.96 by Predicting the Past
Two Class 12 students have three weeks and one shared laptop for their AI capstone. A local NGO wants early warning of tomato late blight. The team is given 1,400 rows from two seasons with 38 columns: rainfall, humidity, temperature, soil pH, village, sowing date, spray_count, harvest_date, yield_loss_percent, and the label outbreak (22% positive).
They train a random forest, obtain 0.96 ROC-AUC on a random 80/20 split, and draft the slide 'Our model predicts blight with 96% accuracy'. Their teacher notices that spray_count is recorded only after farmers report symptoms, and that the NGO's actual need is a warning 7 to 10 days before symptoms appear, so field teams can visit villages. There is no data for the coming season. The NGO's field staff can act on about 10 village-level alerts per week, and they will not trust a system that cries wolf.
1. Audit the project as it stands. Identify the leakage, the evaluation flaw and the mismatch between the model and the NGO's decision.
Read the worked answer
Start with leakage. spray_count is recorded only after farmers report symptoms, so it is a consequence of the outbreak. harvest_date and yield_loss_percent are also realised after the event. Any model using them is reading the answer off the answer sheet. When these columns are removed, the 0.96 will fall, possibly a lot, and the students should expect and report that drop rather than hide it.
The split compounds the problem. A random 80/20 split mixes both seasons and the same villages into train and test, so the model memorises village and season quirks. The honest test is train on season 1 and test on season 2, or grouped by village.
The framing is wrong for the decision. The NGO needs 'which ten villages should we visit next week', not 'what is the probability this row is positive'. A model tuned for ROC-AUC optimises ranking across the whole population; the NGO cares about precision among the top ten alerts, because a week of false alarms destroys trust. The label should also be rebuilt as 'symptoms appear in this village within the next 7-10 days', with features cut off at the decision date.
2. Rewrite the project plan for the remaining three weeks so the NGO gets something usable and honest. Be specific about the sequence of work.
Read the worked answer
Week 1 - fix the table before touching models. Define the decision date t for each row. Keep only features observable at or before t: rolling rainfall, humidity, temperature, soil pH, village, sowing date, days since sowing. Delete spray_count, harvest_date and yield_loss_percent. Rebuild the label as an outbreak in that village in the window t+1 to t+10. Split by season: train on season 1, validate and test on season 2. Fit a logistic regression baseline first, so there is something to beat.
Week 2 - model and evaluate against the decision. Train gradient boosting or random forest on the cleaned table. Report precision@10 per week (of the ten alerts sent, how many villages actually had an outbreak), recall within the top ten, calibration, and the confusion matrix at the chosen cut-off. Do error analysis by village and by week of the season, and check whether the model is simply learning 'it rained a lot'.
Week 3 - documentation and rehearsal. Write the data card, the cut-off rules, the metrics table with the baseline, and the limitations. Build a demo that outputs a ranked weekly list of ten villages, and agree a shadow season with the NGO where alerts are compared with field reports but nobody acts on them yet.
3. Write the outline of the final report and presentation, and state which claims the team must not make.
Read the worked answer
The report should open with the problem as the NGO states it, then the data card: source, two seasons, 1,400 rows, 22% positive, missingness, and the decision date rule. Follow with the feature list and an explicit note on what was removed and why. Then the split strategy, the baseline, the model comparison, and a metrics table anchored on precision@10 and recall within ten alerts, with confidence intervals or at least per-season variation. Include the confusion matrix at the operating threshold. Then limitations, ethics and a validation plan.
The presentation is five slides: the NGO's decision, what the data can and cannot support, the ranked-list demo, one honest metrics slide, and what happens next season. Rehearse the sentence 'we found leakage in our first model, removed it, and here is the lower, real number' - it is the strongest thing they can say.
Claims to avoid: '96% accuracy', any causal statement such as 'spraying reduces loss', any suggestion the model is ready for farmers to act on, and any silent deletion of the leakage discovery. Also avoid a single headline number without the capacity constraint beside it; ten alerts per week is the unit of usefulness.