The 93% Crop Disease Model That Misses Disease
An AI club trains a model to sort tomato leaf photographs into 'healthy' and 'leaf blight'. The training set has 2,000 images collected from three fields in one village on two sunny afternoons, all taken with the same phone. Only 140 images show blight; the other 1,860 are healthy. On a random 20% test split the model reports 93% accuracy, and the club prepares to publish a phone app for farmers. A teacher then tests the app on 30 blighted leaves brought from a different village, and the model labels 24 of them healthy. The mentor points out that the test split was random and that every training photo came from the same three fields. The club has no budget for new data collection this term, but a second phone can be borrowed and the teacher can bring leaves from the new village.
1. Explain how the model can report 93% accuracy while calling 24 of 30 blighted leaves healthy. Support your answer with the numbers.
Read the worked answer
Do the arithmetic before believing the headline. Blight is 140 of 2,000 images, which is 7%. A model that ignores the image entirely and always outputs 'healthy' scores 1,860 divided by 2,000, which is 93%, on any split drawn from the same pool. The reported figure is exactly the score of a useless model, so accuracy cannot separate the two.
The 30-leaf test is more informative. The model caught only 6 of 30 blighted leaves, a sensitivity (recall) of 20%; it missed 80% of disease cases, which is the error that matters, because a farmer acts on 'healthy' by doing nothing.
The gap between 93% and 20% has two causes: class imbalance, which lets the majority class dominate, and distribution shift, since the test leaves came from another village with different soil, leaf age, light and camera. A random split from three fields on two sunny afternoons measures memorisation of those fields, not generalisation.
2. Recommend which metrics and which evaluation split the club should use instead, with reasons.
Read the worked answer
Report a confusion matrix, not one number. For this use, recall for blight (of all truly blighted leaves, how many were flagged) is the primary metric, because a missed case can cost a crop. Precision matters second, because too many false alarms make farmers ignore the app. Add balanced accuracy or macro-F1 so the majority class cannot dominate, and always show the raw counts.
Change the split. Split by group, not at random: hold out an entire field or village, ideally with a different phone and a different day, so the test set is genuinely unseen. Keep a third set that nobody tunes on, otherwise repeated tuning leaks information and the final figure becomes optimistic again.
Finally, evaluate at a chosen decision threshold and tabulate recall and precision at two or three thresholds, so the club can deliberately pick one that favours catching disease rather than accepting whatever the default threshold gives.
3. The club has no money for new data. Propose realistic steps to improve the model and state precisely what claim they may make about it.
Read the worked answer
Spend effort where the error is. The cheapest useful data is more blight: borrow the second phone and photograph affected leaves in the new village, in morning and evening light, at several angles and distances, aiming for a few hundred examples rather than a balanced thousand. Augment the existing set with rotations, brightness changes and blur so the model sees variation the three fields never provided. Use class weights or a lower decision threshold to push recall up, then re-check on the held-out village.
The honest claim is narrow: 'a screening aid trained on 2,000 images from three fields, which flagged only 6 of 30 blighted leaves from a new village in our last test; it is not ready for farmers.' Add the next experiment and the data it needs. Never present 93% as field accuracy, and never let the app say 'healthy' without asking the farmer to confirm visually.