The 97% Model That Failed in the Next Village
Two Class XII students build a leaf-disease classifier for a school AI exhibition. They photograph 640 leaves in their own village over two days: 400 healthy, 160 leaf blight, 50 rust and 30 bacterial spot. All photos use the same phone, the same lighting and the same field. They split the images randomly 70/30 and train a small CNN. Training accuracy reaches 97%, but when a teacher brings leaves from a neighbouring village the model labels most blighted leaves as healthy. On the 30% test set the model scores 62% overall but 88% on the healthy class. The students have one week left, a laptop with 4 GB RAM, and a rule that student photos must not be uploaded to any website. They must present results honestly at the exhibition.
1. Diagnose the gap between 97% training accuracy and 62% test accuracy, and explain the neighbouring-village failure.
Read the worked answer
Two separate problems are being read as one.
Leakage from the random split. The photos were taken on two days in one field, several images per leaf and plant. A random 70/30 split puts near-identical images - same leaf, same background, same light - in both train and test. The test set is therefore not 'unseen'; it is a memory quiz. 97% measures recall of the field, not learning of disease. Correct split: group by plant, then by date or field, so no leaf or plant appears on both sides.
Class imbalance. 400 healthy against 30 bacterial spot means a model can score well by favouring healthy. The 88% healthy accuracy alongside 62% overall is the signature: the majority class is carried, the rare classes collapse.
Domain shift. The neighbouring village has different soil colour, leaf variety, camera angle and light. A model keyed to background and lighting transfers poorly. That is not a bug to hide; it is the honest finding of the project.
State all three in the exhibition poster. A weak but correctly measured model is a better project than a 97% figure that cannot survive a walk to the next village.
2. Which metric should the team optimise, and why is overall accuracy a poor guide here?
Read the worked answer
Optimise recall on leaf blight, reported alongside per-class precision, and summarise with macro-F1.
Why accuracy misleads: with 400 of 640 images healthy, predicting 'healthy' for everything yields about 62.5% accuracy - almost exactly the 62% observed. Overall accuracy can therefore be high while the model is useless for the decision that matters.
Why blight recall: a missed blight case means no treatment and spread to neighbouring plants; a false alarm means an unnecessary spray. The costs are asymmetric, so the threshold should be shifted to favour catching blight, and the resulting precision loss stated openly.
Report a confusion matrix, not one number, and give per-class recall: healthy, blight, rust, bacterial spot. Bacterial spot with 30 examples will be weak; say so rather than averaging it away.
Also report the metric on a field-held-out set, since that is the deployment condition. If blight recall is 60% on leaves from an unseen field, that is the figure to present, with the caveat that more data from other fields is required before any farmer uses it.
3. Propose a one-week plan that fits the laptop and no-upload constraints and still produces defensible results.
Read the worked answer
Days 1–2 - Rebuild the test set. Re-split by plant and date, and reserve a small set of leaves from a second field as an external check. This alone may drop the headline number; that is the point.
Days 3–4 - Train on the laptop. Resize images to 128×128 to fit 4 GB RAM, use a small CNN or transfer learning with a frozen base, batch size 16. Apply class weights inversely proportional to frequency (blight, rust and spot get more weight) and light augmentation - rotation, brightness, slight crop - to simulate field variation, not to duplicate images.
Day 5 - Threshold and evaluate. Tune the blight decision threshold for recall, then record the confusion matrix on both the held-out field and the second-field set.
Day 6 - No-upload compliance. Keep all processing local; if a pretrained base is needed, download weights once on a teacher's connection and document it, but never upload student photos.
Day 7 - Write the poster. Show the honest numbers, the leakage diagnosis, the class counts and the failure on the new village. Recommend more data collection as the next step. Exhibitions reward correct reasoning; a 62% model with a sound evaluation beats a 97% model built on leakage.