The Pilot That Had No Exit Criteria
A financial services firm is piloting an AI-assisted loan document review tool. The pilot was scoped as 'six months, three teams, evaluate results.' At month four, the sponsor reports the tool is 'working well' and requests a full rollout budget of $1.8M. The pilot has no predefined success metrics, no baseline for current review time, and no agreed definition of what 'working well' means. Two of the three teams used the tool heavily; the third barely used it because of a workflow incompatibility. The sponsor argues that delaying rollout risks losing momentum.
1. Evaluate the sponsor's rollout request. What can and cannot be concluded from the pilot as designed?
Read the worked answer
The pilot cannot support a rollout decision because it has no falsifiable success criteria.
What cannot be concluded: that the tool works, that it will deliver value at scale, or that $1.8M is justified. 'Working well' is an impression, not a measurement. Without a baseline review time, there is no way to compute improvement. The third team's non-use is a signal, not noise: it suggests the tool's value is workflow-dependent, which matters enormously for a full rollout.
What can be concluded: two teams adopted it, which indicates usability in some workflows; one team did not, which indicates a fit problem that scaling would multiply.
The sponsor's 'momentum' argument is a reason to decide quickly, not a reason to decide without evidence. The correct response is to convert the pilot into a measurable one before committing $1.8M.
2. Define the exit criteria and decision structure that should have existed at pilot start, and recommend how to recover now.
Read the worked answer
Exit criteria that should have been set at start:
- A baseline metric (median review time per document, error rate) measured before the tool.
- A target improvement threshold and a minimum sample size.
- A definition of adoption (e.g., % of eligible documents processed through the tool).
- A decision rule: roll out if X, extend if Y, stop if Z.
Recovery now: Freeze the rollout decision for 60-90 days. Measure the baseline retrospectively if records allow, or run a controlled comparison (tool vs. no-tool) on comparable document batches. Diagnose the third team's workflow incompatibility explicitly, since it predicts rollout failure modes.
Then present a decision memo with the measured delta and a go/extend/stop recommendation. The $1.8M decision should rest on the measured delta, not on sponsor enthusiasm.