Algorithmic Fraud Triage at Metroville Council

The central concern is distributive justice, or disparate impact. The model's errors are not spread evenly: gig-economy households have a 68% false-positive rate, and two districts holding 14% of households generate 40% of referrals. That means the cost of the system's mistakes - investigation letters, time, stigma and possible loss of benefit while a case is open - falls disproportionately on the poorest residents, while the benefit of fraud detection accrues to the council as a whole. A Rawlsian test would ask whether the least-advantaged group would accept this arrangement behind a veil of ignorance, and the answer is plainly no. A consequentialist framing reaches a similar place once the 20-minute clearance time is included: 68% of a large referral stream is wasted investigator capacity plus harm to innocent households. Procedural justice adds a second violation: with no published appeal route, a flagged household cannot contest a decision it cannot see, and the vendor's refusal to disclose feature weights removes any external check on the scoring logic.

Renewing as-is is hard to defend, because the council cannot demonstrate due diligence on a system whose weights it cannot inspect. Suspension is defensible but forfeits any recoverable fraud and leaves investigators with no triage. The most defensible recommendation is to modify the contract and the workflow, with conditions. Justification: a 32% true-positive rate is not automatically bad - it depends on the counterfactual cost of manual selection - but the disparity between districts and the 68% subgroup false-positive rate are evidence of proxy discrimination, probably through postcode. Modification should include removing or testing proxy features, disaggregated error reporting by income type and district, a published appeal route, and human review before any household is contacted rather than after. The key condition is contractual audit access: if the vendor will not grant the council the right to test the model for disparate impact, the council cannot meet its public-sector equality duties and should suspend. Answers that name a single option but do not state an enforceable condition should be marked down.

Explainability changes the decision in two ways. First, it is a legal and procedural requirement rather than a nice-to-have: households affected by an automated score arguably have a right to a meaningful explanation of a decision about their benefits, and the council cannot provide one if the vendor treats feature weights as confidential. Second, it is a governance tool: without feature-level visibility the council cannot test for proxy discrimination, cannot audit drift, and cannot rebut a complaint. In procurement terms this converts 'proprietary risk score' from an acceptable commercial position into a disqualifying one. A well-argued answer would require, as contract terms, model documentation, per-case reason codes, audit logs, the right to commission independent testing, and a contractual warranty on disparate impact. It should also acknowledge the trade-off: an interpretable model may have lower raw accuracy than a complex ensemble, and the council must decide whether marginal predictive gain justifies the loss of contestability.