The Pilot Portfolio That Outperformed Its Budget and Underperformed Its Promise
A 4,000-person insurer has run eleven generative-AI pilots over fourteen months. Each was sponsored by a functional VP with a $150k-$400k discretionary budget, and the aggregate spend is $2.6M. Nine pilots report 'positive' internal metrics, but only one is in production, and it saves roughly $180k a year. The CFO has frozen new AI funding pending a portfolio review. The CEO wants a three-year AI strategy within six weeks. The Chief Data Officer notes that four pilots touch the same customer records with different consent assumptions. The CIO says the model layer is 'a commodity' and the real blocker is integration capacity: her team has 6 engineers and a 9-month backlog. The CHRO reports that no role descriptions have changed and that managers are routing around the tools.
1. Diagnose why the pilot portfolio produced so little realized value, and identify the two or three root causes that a strategy must address first.
Read the worked answer
The symptom (nine 'positive' pilots, one in production) is a selection and scaling failure, not a technology failure. Three root causes are visible in the facts:
- Sponsorship decoupled from accountability. Each VP spent discretionary budget on a locally-defined success metric. 'Positive internal metrics' were never tied to a P&L line, a customer outcome, or a production commitment. With eleven sponsors and no portfolio owner, nobody was accountable for the transition from pilot to run.
- Integration capacity is the binding constraint, and the portfolio ignored it. Six engineers with a nine-month backlog cannot industrialize eleven pilots. A strategy that funds more models without funding integration, data contracts, and platform work will reproduce the same result. The CIO's 'commodity' framing is half right: differentiation is in integration and data, not in model choice.
- Unmanaged cross-cutting risk. Four pilots touching the same customer records under different consent assumptions is a governance defect that will surface as a compliance incident the moment any of them scales. This is a reason to consolidate, not to pause everything.
A useful reframe for the CEO: the $2.6M bought learning, which is legitimate, but the portfolio lacked a kill/scale gate. Recommend a portfolio triage (keep, merge, kill) against three criteria - quantified value at production scale, integration cost, and risk exposure - and fund the integration backlog as a first-class line item before approving new pilots.
2. The CEO asks for a three-year strategy in six weeks. What should the strategy contain, and what should it deliberately not contain, given the constraints?
Read the worked answer
Six weeks is enough for a decision-shaped strategy, not a comprehensive one. It should contain:
- A value thesis tied to 3-4 named P&L or customer outcomes, each with a baseline, a target, and an owner at VP level. 'Reduce claims-cycle time by X days' beats 'deploy AI across operations.'
- A capacity plan that treats integration and data engineering as the rate limiter. State explicitly how many production use cases 6 engineers can support per year, and either fund more capacity or cap the portfolio. A strategy that assumes unconstrained engineering is fiction.
- A governance spine: one consent and data-use standard for customer records, a model inventory, and a review gate that applies to pilots from day one - cheap to run at eleven pilots, impossible to retrofit at fifty.
- A talent and role-change plan, since no role descriptions have changed and managers are routing around tools. Adoption is a design problem, not a communications problem.
- A staged funding model with explicit kill criteria and a portfolio review cadence.
It should not contain: a technology vendor roadmap, an aspirational 'AI-first' slogan, a list of every possible use case, or three-year financial projections precise to the decimal. Those create false confidence and will be wrong. Present the strategy as a set of bets with stated assumptions and the evidence that would change them.
3. The CFO proposes killing all pilots except the one in production and reallocating the budget to the integration backlog. Evaluate this proposal.
Read the worked answer
The instinct is sound but the proposal is too blunt in one direction and too generous in another.
Where the CFO is right: stopping net-new pilot spend until integration capacity exists prevents repeating a $2.6M pattern. Funding the integration backlog is the highest-leverage move available, because it unblocks every future use case.
Where it is too blunt: some pilots may be one integration away from material value, and killing them destroys sunk learning and sponsor goodwill. A triage is better than a blanket cut - score each pilot on value at scale, integration cost, and risk, and keep the two or three with the best ratio. Also, the four pilots sharing customer records should be merged into one governed initiative rather than killed, because their overlap is exactly the consolidation opportunity.
Where it is too generous: the surviving production use case ($180k/yr) should be held to a real run-cost standard - inference, maintenance, and support costs are rarely in the pilot numbers, and a $180k saving can evaporate against $120k of run cost plus engineering time.
Recommendation: approve the integration investment, run a two-week triage, keep a small protected exploration budget (e.g., 10-15% of AI spend) so the organization does not lose the capability to evaluate new options, and attach a named owner and a production date to anything that survives.