The Adjuster Who Always Agreed
A motor insurer deploys an AI claims-triage system. It auto-approves 65% of simple claims and routes the remaining 35% to 120 human adjusters with a recommended decision and a confidence score. After six months, average handling time is down 41% and adjusters accept the AI recommendation 88% of the time. A sample audit finds that complex claims where the AI said 'approve' were three times more likely to be wrong than complex claims decided by adjusters without an AI recommendation. Adjusters say they scrutinise 'when the score is low'; their training was a 45-minute video. The regulator has asked for evidence that human oversight is genuine, and the company wants to raise auto-approval to 85%. The model is a gradient-boosted tree trained on three years of claims; its confidence score is not calibrated.
1. Explain automation bias and show how the audit evidence indicates that human oversight is not genuine.
Read the worked answer
Automation bias is the tendency to accept automated recommendations more readily than human ones, especially under time pressure, when the automation is presented as authoritative, or when the human is not given independent evidence. The 88% acceptance rate is not itself proof of bias, because the AI may often be right, but the audit is: on complex claims, AI 'approve' recommendations were three times more likely to be wrong than adjuster-only decisions. That suggests adjusters are not adding independent judgement on exactly the cases where it matters. The 45-minute video and the confidence score anchor the review; 'scrutinise when the score is low' delegates the decision to the model's own uncertainty estimate, which is uncalibrated. Genuine oversight requires the human to have the authority, time, information, and incentive to disagree. Here, handling time fell 41%, which may mean less review, not better review. The pattern is consistent with rubber-stamping: the human is 'in the loop' formally but not substantively.
2. Design a human-AI oversight model that would satisfy the regulator.
Read the worked answer
Move from blanket review to risk-based, evidence-based oversight. (1) Segment claims by consequence and uncertainty: high-value, complex, vulnerable-customer, and low-confidence cases require independent review; simple low-value cases may be auto-approved with audit. (2) Give adjusters the underlying evidence and a short list of contributing factors, not only the recommendation and score; ask them to record a reason for agreement or disagreement. (3) Use 'consider the opposite' prompts and periodic blind cases where the AI recommendation is hidden, to measure adjuster accuracy independent of automation. (4) Calibrate the confidence score and validate it on the live population; an uncalibrated score is a poor routing rule. (5) Set override targets as diagnostics, not penalties: a very low override rate is a warning, not a success. (6) Sample audits by segment, with feedback and retraining. (7) Log decisions, preserve accountability, and give customers an explanation and appeal route. These measures make oversight testable rather than nominal.
3. Should auto-approval rise from 65% to 85%? Advise the board.
Read the worked answer
Not on the current evidence. Raising auto-approval from 65% to 85% shifts more decisions to the model precisely when the audit shows complex AI approvals are less reliable. The board should first separate claim segments: if the additional 20 points are simple, low-value, well-calibrated claims, a controlled increase with audit may be reasonable; if they include complex or high-consequence claims, it is not. Set gates: segment-level precision and recall, complaint and appeal rates, wrongful-denial rates, and a holdout sample of human review. Run a phased increase (for example 65% to 70% to 75%) with monthly monitoring and a kill switch. Keep human review for vulnerable customers and edge cases. Also address the workforce effect: adjusters' role should shift to exception handling and oversight, which requires training and realistic workload. The regulator is asking for evidence of genuine oversight; expanding before fixing it would weaken the company's position. The goal is not maximum automation but the lowest total harm at acceptable cost.