The GenAI Pilot With No Owner
A 400-person logistics firm runs a 10-week generative AI pilot: 60 employees use an assistant to draft customer emails, summarize carrier contracts, and write SQL against the operations warehouse. Adoption is strong (48 daily active users by week 6). Problems emerge: an employee pasted a carrier's confidential rate sheet into the tool; a SQL query written by the assistant dropped a staging table because it inferred 'clean up' from a vague prompt; and two teams built overlapping tools because no one tracked who was doing what. There is no AI policy, no acceptable-use guidance, no logging of prompts, and no named owner. The CIO asks whether to expand to 250 users next quarter.
1. Identify the governance gaps and rank them by risk.
Read the worked answer
Ranked by risk (highest first):
- Confidential data exposure. A carrier rate sheet went into a third-party tool. This may breach the carrier contract and data-processing terms. Unknown whether the vendor trains on inputs or retains them. Immediate action: confirm vendor data-handling terms, assess notification obligations, notify the carrier if required.
- No acceptable-use policy or training. Employees are guessing what is allowed. This caused #1 and will cause more.
- Destructive-action risk. The assistant wrote SQL that dropped a table. There is no guardrail separating read from write access, and no human review of generated SQL.
- No ownership. No named accountable person means no one is deciding, no one is monitoring, and duplication flourishes.
- No logging. Without prompt/response logs you cannot investigate incidents, measure usage, or detect misuse.
- Duplication. Two overlapping tools waste effort and create inconsistent outputs.
The top three are active harm; the bottom three are structural enablers of future harm.
2. Draft the conditions the CIO should require before expanding to 250 users, and say what you would allow in the meantime.
Read the worked answer
Conditions before expansion
- Named owner (e.g., Head of Data or COO) accountable for the program, with a small cross-functional group (IT, legal, security, ops).
- Acceptable-use policy covering: what data may never be entered (confidential third-party data, PII), approved tools only, and consequences. Short, plain-language, mandatory training.
- Vendor due diligence on data retention and training use; contract terms that prohibit training on our data where possible.
- Technical guardrails: read-only warehouse credentials for generated SQL, a review step before any write/DDL executes, and a sandbox for experimentation.
- Logging of prompts and outputs, retained per policy, with access controls.
- A single intake process so new use cases are registered and duplication is caught.
- Success metrics defined up front (time saved, quality, incident count).
What I'd allow now: continue the 60-user pilot under an interim policy (no confidential data, read-only SQL, logging on), because stopping entirely loses learning and pushes use underground. Expansion to 250 waits until conditions 1–4 are met.