FasalMitra: 88% at home, 52% one village away, six weeks to a pitch
Three Class XII students build 'FasalMitra', a crop-disease classifier for their school's vocational exhibition. They photograph leaves in their own village and collect 620 images across five classes: 340 of bacterial leaf blight, 41 of a nutrient-deficiency class, and 239 spread over three other classes. They resize every image to 224x224 pixels, fine-tune a small pretrained CNN on a random 80/20 split, and report 88% test accuracy. When their teacher brings 60 photographs from a neighbouring village, accuracy falls to 52% and most deficiency leaves are labelled as blight. The students plan to pitch to the district agriculture officer in six weeks, charge Rs 30 per farmer per month, and run the model offline on phones with 2 GB RAM and patchy 3G; the current model file is 45 MB. Board examinations begin in five weeks. Their only labelled data is what they photographed themselves, and their pamphlet recommends a specific fungicide dose for blight.
1. Explain the 88% versus 52% gap. What does it tell you about their data, their split and their metric?
Read the worked answer
The split is not independent. Photographs taken in a burst of the same leaf, on the same plant, in the same light, end up on both sides of a random 80/20 split. The test images are near-duplicates of training images, so 88% measures memorisation of that village's leaves, not recognition of disease. A grouped split (all images of one plant or one farm in a single fold) would already have lowered the score.
The data are not representative. One village, one phone camera, one season, one set of soil and lighting conditions. The model latches onto whatever correlates with the label in that setting, which may be background, leaf colour cast or time of day rather than lesion shape. The neighbouring village shifts all of those, hence 52%.
Class imbalance explains the specific failure. With 340 blight images and 41 deficiency images, a model that never predicts deficiency still scores well on a blight-heavy test set. So deficiency leaves are called blight, exactly what was observed. The consequence is not academic: deficiency needs nutrients, blight needs fungicide, so the confusion leads to the wrong intervention and wasted money.
The metric is wrong for the problem. Accuracy under imbalance rewards the majority class. They need the confusion matrix, per-class recall (especially deficiency), and macro-F1, plus a leave-one-village-out estimate as the realistic generalisation figure. Reporting 88% to a district officer would be a claim their own 60-image test already contradicts. The honest headline is: works on our village, does not yet transfer.
2. Propose a six-week plan up to the pitch and exhibition that fits around board examinations. State what the MVP is, what data and evaluation work comes first, and what they must not ship.
Read the worked answer
MVP definition. A free, offline, on-device tool that names the two most likely causes with a confidence level and an explicit 'not sure, ask your Krishi Vigyan Kendra' option. Not a diagnosis engine, not a prescription engine, and not a paid product yet.
Weeks 1-2 (data, before any modelling). Collect images from at least three villages, at least three phone models, healthy and diseased leaves, morning and afternoon light, and both young and old lesions. Label with an agronomist or a KVK officer present, so the deficiency class is trustworthy. Target at least 150 images for the weakest class or merge it into a broader 'not blight' category. Fix the split: group by farm or village and report leave-one-village-out. Do not tune anything until the split is fixed.
Week 3 (model and size). Train a smaller backbone, quantise to under 10 MB, and measure latency on an actual 2 GB phone in airplane mode. Target under 3 seconds per photo and a download that works over 3G. If 45 MB does not shrink, the offline promise fails.
Week 4 (evaluation). Report macro-F1, per-class recall, the confusion matrix and a calibration check. Set a threshold: below it, the app abstains. The key number for the pitch is the leave-one-village-out score, not the random-split score.
Week 5 (field test). Ten farmers photograph their own leaves; compare the app's answer with the agronomist's. Measure how often it is right on their photos and how long it takes. Board exams sit here, so cap the work and split roles: one on data, one on the app, one on the write-up.
Week 6 (pitch). Present with honest numbers, including the abstain rate and the failure cases.
Must not ship: the fungicide dose. Remove it from the pamphlet and replace it with 'confirm dose and product with your KVK or block agriculture officer'. Minors publishing pesticide dosages for a model with a 52% transfer score is an unacceptable risk to crops and to health.
3. The team still wants to charge Rs 30 per farmer per month. Advise them on the business case, the liability, and what they should actually ask the district agriculture officer for.
Read the worked answer
The business case does not yet support a price. Rs 30 per month needs a farmer to believe the tool saves more than Rs 30 of loss. The team cannot show that: their transfer accuracy is 52%, the deficiency class is unreliable, and they have no evidence of yield or cost saved. Charging now also invites refunds and reputational damage in a small district where word travels. A free pilot with 20-30 farmers, measured against the agronomist's diagnosis, produces the evidence that a price later depends on.
Liability and ethics. If a farmer sprays the wrong fungicide on a wrong diagnosis, harm falls on the crop, the household income and possibly health. The students are minors with no agronomic qualification and no insurance. Two protections: never output a product or dose, only a ranked possibility plus an abstain option; and route every recommendation through a qualified human (KVK, block agriculture officer) whose contact is in the app. They should also handle photos responsibly: capture on the device, upload only with the farmer's permission, and say plainly what is stored.
A more realistic model. Sell to the institution, not the individual: a free app for farmers, with a paid or grant-funded dashboard for cooperatives, FPOs or the agriculture department, who value the aggregate picture of disease spread. That avoids collecting Rs 30 from households with irregular cash flow and puts the payer where the data value sits.
What to ask the district agriculture officer for. Not an endorsement of the app, but: access to labelled data and an agronomist's time for validation; permission to run a supervised pilot in two blocks; feedback on which classes actually matter locally; and a letter of association that the team can quote honestly. In return, offer the pilot data and a report. The pitch should open with the 52% figure and the abstain rate, because an officer who discovers the gap later will not trust anything else the team says.