Quantise on the Pi, or Send the Photo to the Cloud?
A cooperative wants to run a leaf-disease and soil-moisture advisory on a Raspberry Pi 4 (2 GB) in a tubewell shed. The trained model is 45 MB, takes 1.4 s per image on the Pi, and must answer within 1 s of a photo being taken; the same Pi also runs a camera capture loop and a local web page. Power comes from a 20 W solar panel and a battery, and 4G drops for hours during the monsoon. Two options are on the table: prune and INT8-quantise the model (expected 41% smaller, about 2.6× faster, validation accuracy 94.1% → 92.8%), or send each image to a cloud endpoint (3–6 s round trip, about ₹0.4 per image, needs 4G). The cooperative handles 300 photos a day and cannot revisit farmers' fields to recheck mistakes. The device must keep working through a three-day outage.
1. Choose between on-device quantisation and cloud inference, supporting the decision with arithmetic on latency, cost, memory and connectivity.
Read the worked answer
Choose on-device INT8 quantisation, conditional on a per-class check.
Latency: cloud is 3–6 s against a 1 s requirement - a hard fail before any other factor. Quantised on-device inference is roughly 1.4 / 2.6 ≈ 0.54 s, leaving about 0.46 s of headroom for capture, resizing and the web page. That headroom matters because the Pi also runs other work.
Memory: 45 MB × 0.59 ≈ 27 MB, comfortably inside 2 GB, and a smaller working set reduces pressure on the capture loop.
Cost: 300 × ₹0.4 = ₹120 a day, roughly ₹3,600 a month, every month, plus a data plan - for a cooperative that is a recurring bill for a worse service.
Connectivity and power: 4G fails for hours in the monsoon, precisely when disease pressure rises; cloud inference would be dead during the outage. On-device keeps a 20 W panel and battery self-sufficient, and shorter inference means less energy per photo.
Accuracy trade: 94.1% → 92.8% overall is 1.3 points. Accept it only if blight recall does not fall more than about 2 points; rare classes can be hit hardest by quantisation. Run that check before rollout.
2. Design the monitoring and rollback plan for the deployed quantised model.
Read the worked answer
Log on device: timestamp, predicted class, confidence, image hash, battery level, panel voltage and device temperature. Store locally and sync when 4G is available; never block inference on the network.
Weekly audit: a staff member reviews 30 randomly selected stored images with the farmer's reported outcome, and records disagreements. Watch the confidence distribution - a sudden drift toward low confidence usually precedes visible accuracy loss.
Alerts: class-mix shift (for example blight jumping from 15% to 40% of predictions), confidence median falling below the training baseline, or temperature throttling that pushes latency above 1 s.
Rollback: keep the original float model on the SD card under a versioned name, and make switching a one-line configuration change, not a re-flash. Document the procedure in Punjabi and English for the shed operator.
Shadow period: run the quantised model alongside the float model for two weeks, log both predictions, and compare before trusting either.
Seasonal recheck: repeat the audit after the first monsoon weeks, since lighting, leaf wetness and phone camera angles all change, and a model validated in dry weather is not automatically valid in rain.
3. What specifically could go wrong with INT8 quantisation, and how would you test for it before trusting the deployment?
Read the worked answer
Likely failure modes
- Calibration mismatch. INT8 needs a representative calibration set. If it contains mostly bright, dry-season images, the scale factors are wrong for dim or wet leaves and activations clip. Build calibration from real field photos across lighting, angles and all four classes.
- Rare-class collapse. Quantisation error is not spread evenly; bacterial spot with 30 training examples can lose most of its recall while overall accuracy moves only 1.3 points. The average hides it.
- Per-tensor vs per-channel. Per-tensor scaling on convolutional weights often costs more accuracy; use per-channel for weights.
- Thermal throttling. The Pi may hit the 1 s budget after twenty minutes in a hot shed even if it passes on the bench.
Test plan: hold out 50 field images never used in training or calibration. Compare float vs INT8 per class - recall, precision and confusion matrix - and require blight recall to stay within 2 points. Run a one-hour sustained loop and record latency percentiles, not the mean. Test low light and motion blur deliberately. Reject the quantised build if any rare class loses more than 5 recall points, even if overall accuracy looks acceptable.