Adventure Works: parcel defect detection and label reading
Adventure Works Logistics photographs 4,000 parcels a day across 12 conveyor lines. Two goals: flag crushed or torn parcels so a human repacks them before they reach downstream equipment, and read the printed SKU plus the handwritten route code on each label. Their current attempt uses a whole-image custom classification model trained on 300 labelled photos from one site. It reports 71% accuracy overall but 38% on the night shift, where lighting is dimmer and shrink wrap throws reflections. A missed tear is expensive: it damages sorters. They also ran OCR on 640 px thumbnails and got 62% character accuracy on handwriting. Peak volume is 900 parcels per hour per line.
1. Choose the right vision capability for each of the two goals and justify the choices against the failure modes described.
Read the worked answer
Defect detection → object detection The question is not 'does this image contain a defect' but 'is there a tear, where, and is it large enough to repack'. A parcel can carry zero, one or several defects; bounding boxes give both the decision and the evidence a reviewer needs. Whole-image classification can only answer a yes/no for the entire frame and cannot localise, so a reviewer cannot verify a flag - and 71% overall tells you it cannot even do the yes/no reliably.
Tradeoff to state: if every parcel is centred in its own frame and can carry at most one defect, classification on the crop is cheaper to label and train. Their framing does not guarantee that.
Label reading → the Read/OCR capability, not the custom model Text extraction is a different product: use Azure AI Vision Read (or Document Intelligence prebuilt-read). It handles printed and handwritten text, but handwriting accuracy depends heavily on resolution and contrast - 640 px thumbnails are the direct cause of 62% character accuracy. Feed native-resolution images and crop to the label region, which raises character height and removes background noise.
Better architecture: two stages Detect the label region (or use a fixed region of interest per camera), deskew, then OCR the crop. One full-frame OCR call per parcel wastes resolution on the background and drags accuracy down.
2. With only 300 labelled images and a 38% night-shift result, what is the highest-value fix?
Read the worked answer
The night-shift gap is a data mix problem, not a model-capacity problem. 300 images from one site, presumably mostly daytime, means the model has barely seen glare, low contrast or motion blur. Throwing a bigger model at it will not help.
Highest value, in order
- Stratify collection: label a balanced set across site × shift × lighting × parcel type; a slice that has 2% of the data cannot reach acceptable accuracy. This is the single change that moves the 38%.
- Balance negatives: include plenty of clean parcels, otherwise the model learns to flag everything and reviewers drown.
- Augment for the conditions actually observed - brightness/contrast shifts, glare spots, blur, JPEG artefacts. Do not augment beyond label truth (a subtle rotation is fine; a synthetic tear that is not in the box is not).
- Fix labels before training: with 300 images, a few inconsistent boxes cap accuracy. Write a labelling guide and double-annotate a sample to measure inter-annotator agreement.
- Evaluate per slice, not in aggregate. '71% overall' hides a 38% slice. Gate promotion on the worst slice, and set review thresholds per slice.
On volume: object detection typically needs more than 300 images, so the answer combines a labelling plan with the architecture - every human review decision becomes a free labelled example, so the review queue is the source of training data, not a dead end.
3. Design the production pipeline for peak load of 900 parcels per hour per line, including how confident decisions become actions.
Read the worked answer
Throughput maths first. 900/hour/line ≈ 0.25 parcels/s per line; 12 lines ≈ 3/s average with peaks well above that. Per-image synchronous calls from the conveyer controller will not survive peaks and will couple line stoppage to API latency.
Decoupled pipeline Image → Blob storage → queue → worker (Functions or container) → vision batch/asynchronous API → results queue → review UI and line controller. The controller reacts to a result, never to a call it must wait on. Use the service's batch/async path for high volume and check the standard-tier transaction limits against peak.
Three-tier decision with thresholds
- High confidence defect → auto-flag, repack.
- Mid band → human review; never block the line, divert to a review lane.
- High confidence clean → auto-accept, but sample-audit 1-2% of auto-accepts. Because a missed tear damages equipment, the cost function is asymmetric: tune toward recall and accept more false positives.
Operational detail
- Choose a region that is available, close to the sites, and acceptable for image residency.
- Version models; shadow-run a new model on live traffic before promoting it.
- Feed reviewer corrections back monthly; the retraining loop is what keeps night-shift performance from silently decaying as cameras drift.
- Monitor per-slice accuracy in production, not just aggregate accuracy - the night shift is where it will regress first.