Fraud Risk Model Flagging Social Security Claims: Suspend, Shadow-Run or Continue?
You are chief data officer in a social security agency that pays 2.1 million claims a year. A machine-learning fraud risk model, bought from a vendor, flags 6.2% of claims for review. Internal audit tested 300 flagged cases and found 18% were wrongly flagged, while the vendor reports 92% accuracy. The model uses postcode, age and benefit type; the legal team warns it may indirectly discriminate and has not completed a data protection impact assessment. Since go-live, 1,100 claims have been suspended, and two media outlets have asked for comment. The model documentation is incomplete, and staff in local offices say they are overriding flags without recording why. The minister wants 'the facts, not a shutdown'. The model has been live for five months.
1. What is your assessment of the evidence, and would you suspend the model? Justify your decision.
Read the worked answer
The vendor's 92% is not a sufficient basis for confidence. Accuracy depends on the base rate and on what is counted. If 6.2% of 2.1 million claims are flagged, that is about 130,000 reviews a year. The audit sample of 300 found 18% wrongly flagged; if representative, that is roughly 23,000 false positives a year, with wide uncertainty because the sample is small. The incomplete documentation and unrecorded overrides make the model unauditable. I would not continue automatic suspension. I would suspend the suspension decision, not necessarily the model: keep it in shadow mode or as a triage aid with human decision-making, require a completed data protection impact assessment and equality analysis, and test performance by postcode, age and benefit type. The legal risk of indirect discrimination is serious and cannot be managed by assurances. The minister wants facts: give the error rate, the number affected, the legal exposure and a dated remediation plan.
2. How would you design an evaluation and monitoring framework for the model?
Read the worked answer
Evaluate outcomes, not just model metrics. Define: fraud and error detected, false-positive rate, time to decision, appeals upheld, claimant hardship and staff override rate. Establish a baseline and, where feasible, a shadow-mode comparison or a controlled pilot. Report the full confusion matrix, not accuracy alone, and disaggregate by subgroup. Monitor drift monthly: input data quality, flag rate, override reasons and decision outcomes. Require override logging with reasons, and audit a random sample quarterly. Commission independent validation before any return to automated action. Set thresholds for pause: if false positives exceed an agreed level or subgroup disparity is material, the model stops. Publish a summary. The framework should also cover vendor access, data retention and security.
3. How would you handle the media and ministerial requests?
Read the worked answer
Acknowledge, correct, explain, and protect claimant data. I would coordinate with legal and communications, confirm the facts we can stand behind, and avoid speculation about individual cases. Public line: the agency has paused automatic suspensions, is reviewing affected claims as a priority, has commissioned an independent check, and will publish the assessment. Correct the vendor's accuracy claim carefully by explaining base rates and the audit finding, without turning it into a blame exercise. Offer a technical briefing to journalists and a private briefing to the minister. Prepare Q&A on numbers, legal basis, remedies and timeline. Do not disclose personal data. If errors caused hardship, set up a fast redress route and say so. Silence would allow the story to be framed by the vendor and the opposition.