Insurer's Pilot to Platform: Governance Without a Bottleneck
An insurer with 4,000 employees ran a three-month generative AI pilot with three teams. Results are mixed: 12% weekly active users, two teams report meaningful time savings, but legal discovered engineers pasting claims data into a public chatbot, finance found one team's Vertex AI spend grew eightfold in a single week with no alerting, and departments are now procuring their own tools independently. The CIO has budget for a small platform team and a mandate to show measurable value within two quarters. The CISO wants VPC Service Controls, data loss prevention, and an approved model list. Business units argue that a central review board would kill momentum - the previous pilot took six weeks to approve a single use case.
1. Design a governance model that satisfies the CISO without recreating the six-week approval bottleneck.
Read the worked answer
Hub-and-spoke, with risk tiering as the release valve.
- Hub: a small central platform team owning the guardrails and the paved road - approved model list, reference architectures, prompt and eval templates, the logging pipeline.
- Spokes: named champions inside each business unit who own discovery and delivery, accountable to their own leadership for outcomes.
Risk tiering is what fixes the bottleneck. Define three tiers up front. Tier 1 (internal summarization, drafting on non-sensitive data): self-serve with a checklist and automatic logging - no board. Tier 2 (customer data, regulated content): platform review within five business days against a fixed checklist. Tier 3 (decisions affecting customers, automated actions): full review with compliance and model risk. Publish the criteria so teams self-classify.
Controls: per-project quotas and budget alerts at the billing account, an enterprise tenant as the sanctioned general chat tool, VPC Service Controls perimeter plus CMEK for regulated data, DLP scanning on ingress, and central logging to BigQuery with retention for audit. Enforce the approved list through org policy so shadow procurement fails technically, not just procedurally - a policy nobody can bypass beats a policy everyone is asked to follow.
2. How should the CIO measure value and decide which pilots to scale or kill?
Read the worked answer
Measure the work, not the seats.
- Leading indicators: weekly active users, queries per active user per week, breadth of use cases per team. 12% weekly active users after three months is the real headline - usage decay usually means the tool is not embedded in a workflow, not that the tool is bad.
- Outcome metrics, defined before launch against a baseline: cycle time per claim or quote, rework and error rate, first-contact resolution, queue backlog. Where a clean baseline does not exist, instrument for four weeks first rather than accepting self-reported savings, which run high.
- Unit economics: cost per resolved task or per document processed, including review time, compared with the manual baseline.
- Guardrail metrics: incident count, refusal/false-positive rate, DLP hits, spend variance week over week.
Decision rule, written in advance: scale if the outcome metric moves beyond a stated threshold with guardrails stable; iterate if usage is low but the workflow fit is credible; kill if the task turns out to be low-volume or the human review burden cancels the gain. Report a portfolio view quarterly - a small number of scaled wins plus a pipeline beats twelve stalled pilots.
3. Sequence the first 90 days. What has to be true at day 30, 60, and 90?
Read the worked answer
Days 1-30 - stop the bleeding and establish the baseline. Inventory every tool in use and shut down unsanctioned ones with a sanctioned alternative ready the same day. Publish the approved model list and the risk tiering. Stand up budget alerts and per-project quotas at the billing account. Reconstruct the eightfold spend spike to learn what monitoring failed. Interview the two successful teams and write down what actually worked, because that is the template.
Days 31-60 - build the paved road. Reference architecture with VPC-SC, CMEK, DLP, and logging. Prompt and evaluation templates. Five-day service level for Tier 2 reviews with a published checklist. Nominate champions per business unit. Select two or three use cases with measurable baselines and named business owners, and build evaluation sets before building features.
Days 61-90 - run and report. Weekly metrics review with business owners; shadow deployments before any customer-facing change; a monthly forum where teams show results and failures. At day 90 the CIO should be able to say: spend is bounded, data flows are controlled, and N use cases have measured outcomes with a scale/kill recommendation. If approval time has not come down, the tiering is misdesigned - fix it before adding use cases.