The Automation Business Case With a Human Denominator
Lattice Financial Services plans to deploy an AI document-review tool across its 300-person claims operation. The vendor claims 40% handling-time reduction and 96% accuracy. A six-week pilot on 4,000 claims showed 31% time reduction on straightforward claims but a 9% increase on complex ones, and accuracy of 91% overall - with errors concentrated in handwritten and scanned documents. The tool flags low-confidence cases for human review, which the pilot found occurred on 18% of items. Staff are anxious; the union has asked for consultation. The regulator requires that customers receive reasons for decisions and that firms can demonstrate human oversight of automated decisions. The board wants a decision within a month to capture a vendor discount.
1. Evaluate whether the pilot evidence supports the business case as presented, and identify the analysis the board still needs.
Read the worked answer
The headline 40% claim is not supported.
- Blended effect is not 31%: complex claims got slower by 9%, and 18% of items need human review, which adds handling time rather than removing it. The true net effect depends on the mix of straightforward versus complex claims, which the pilot does not state.
- Accuracy of 91% is below the vendor's 96% and, more importantly, is an average. Errors concentrated in handwritten and scanned documents mean the error rate for that segment is far higher - and those are exactly the customers most likely to be vulnerable or to dispute outcomes.
- A 4,000-claim pilot over six weeks may not capture seasonal mix; claims complexity varies.
Analysis still needed
- Segment-level economics: time saved per claim type × volume per type, minus review time on the 18% flagged, minus rework on the 9% error rate. Build the case bottom-up, not on an average.
- Error cost: cost of correcting a wrong decision, including complaint handling, redress and regulatory consequence. At 9% error on the complex segment, this may exceed the time saving.
- Quality-adjusted throughput: the case should be measured in correctly handled claims per hour, not claims per hour.
- Human oversight cost: the regulator requires demonstrable oversight and reasons for decisions. Reviewing 18% of items is a permanent staffing requirement, not a transition cost; it must be in the run-rate.
- Change and consultation cost: redeployment, training, union consultation time.
Conclusion: the evidence supports a targeted deployment on straightforward claims, not a blanket rollout. The board should be told the vendor discount is a deadline designed to prevent exactly this analysis.
2. Recommend how the board should handle the people, regulatory and timing dimensions of the decision.
Read the worked answer
People
- The union's consultation request is legitimate and, if handled badly, will cost more than the vendor discount. Consult before announcing, with a clear statement of what changes for staff: redeployment, retraining, and no compulsory redundancies in the first phase if that can be honoured.
- Anxiety is rational: staff know the tool performs worse on the cases they find hardest. Involve experienced claims handlers in designing the review workflow - they will spot failure modes the pilot missed.
Regulatory
- Human oversight must be real, not nominal. Flagging 18% for review is only adequate if reviewers have the information and authority to overturn. Document the oversight design, the reasons-for-decision process, and how errors are detected and remedied.
- The handwritten/scanned error concentration is a conduct risk: those customers may be disproportionately disadvantaged. Consider excluding that segment until accuracy improves, or routing it to human-first handling.
Timing
- A one-month deadline set by a vendor discount is a commercial lever, not a governance requirement. The board should separate the decision to proceed in principle from the decision to sign.
- Approve a phased rollout on the straightforward segment now, with a gate review after one quarter using pre-agreed metrics (correctly handled per hour, error rate by segment, complaint rate). Defer the full rollout and the discount decision to that gate.
- Ask the vendor to hold the discount against a signed letter of intent - if they will not, the discount was not real.
Judgement: the board should show it can say 'yes, in part, on evidence' rather than choosing between the vendor's timetable and indefinite delay.