Facial Recognition in the Frozen Aisle
A 62-store supermarket chain runs an 8-week pilot of computer vision plus facial recognition in 10 stores to flag known shoplifters and 'suspicious behaviour'. The pilot produces 1,200 alerts; 140 are confirmed thefts, 46 lead to wrongful stops (31 involving customers under 25), and 8 formal complaints follow, including a viral video. Shrink in pilot stores falls 9% versus 2% in control stores. The vendor cites 98.5% accuracy, but that figure comes from a balanced benchmark of faces, not store CCTV. The legal team notes that biometric data triggers a data protection impact assessment and that local law requires explicit consent or a substantial public-interest basis. Staff already override 18% of alerts, and the board wants a national rollout in six weeks.
1. Using the pilot figures, assess the benefits and risks of this system. Distinguish technical performance from operational and ethical risk.
Read the worked answer
Operationally, the headline benefit is a 9% shrink reduction versus 2% in control stores, a 7-point difference; deterrence may also matter, though the pilot cannot separate deterrence from detection. The alert data tell a different story: 1,200 alerts produced 140 confirmed thefts, so precision is about 11.7% and roughly 1,060 alerts were not confirmed thefts; 46 wrongful stops are a direct harm. The vendor's 98.5% accuracy is not evidence of fitness because it comes from a balanced face benchmark, whereas store CCTV has low prevalence, variable angles, and demographic imbalance. 'Accuracy' also hides the cost of false positives. Technical risk includes benchmark mismatch and bias; operational risk includes alert fatigue (18% override) and inconsistent staff response; ethical and legal risk includes biometric privacy, consent, disproportionate impact on under-25s, reputational damage, and function creep. The viral video shows that a single false stop can impose costs far beyond the pilot's shrink savings. Benefits are real but must be weighed against these harms.
2. Recommend a board decision. State conditions that would have to be met before any expansion.
Read the worked answer
Do not approve a national rollout in six weeks. Approve a pause and a structured re-pilot with gates. First, complete a data protection impact assessment and establish a lawful basis; if explicit consent is impractical, document the substantial public-interest test and necessity/proportionality. Second, commission an independent bias and false-positive audit on store CCTV, reporting precision, recall, and wrongful-stop rates by age, gender, and ethnicity, not a single accuracy figure. Third, change the operating model: alerts should trigger a human review of evidence, not a stop or confrontation; staff need training and a scripted redress process. Fourth, set quantitative gates: precision and wrongful-stop thresholds, complaint rates, and a maximum override rate that signals trust or usability problems. Fifth, run a stepped-wedge or cluster pilot with control stores and pre-registered metrics. Consider less invasive alternatives such as targeted guarding, product tags, or exit gates. The board should treat the 9% shrink result as promising but confounded, not as a mandate.
3. Which risks would remain even if the vendor raised accuracy to 99.9%? Explain why.
Read the worked answer
Better accuracy reduces false positives but does not remove foundational risks. (1) Base-rate problem: when theft is rare, even a highly accurate classifier can produce many false alerts relative to true ones; precision depends on prevalence, not accuracy alone. (2) Privacy and consent: continuous biometric scanning of all customers is a surveillance practice with a chilling effect, independent of error rate. (3) Bias and disparate impact: a model can be accurate overall yet worse for a subgroup; the 31 under-25 wrongful stops are a warning. (4) Legal legitimacy: a lawful basis and proportionality are separate from technical performance. (5) Automation and staff behaviour: staff may over-trust alerts or apply them inconsistently; a 99.9% system can still cause a wrongful stop. (6) Function creep: infrastructure built for theft can be repurposed. (7) Accountability: who explains and compensates a false stop? Accuracy is necessary but not sufficient; governance, redress, and scope limits are also required.