The Hallucinated Compliance Clause
You are the Product Manager for a B2B expense-management product. Over two iterations your PO has used a general-purpose LLM to draft 40 stories and their acceptance criteria. Review found: three acceptance criteria citing a 'SOC 2 Type II quarterly attestation clause' that does not exist in your actual obligations; two stories containing fragments of a real customer's transaction descriptions copied from a support ticket; and a defect rate on AI-drafted stories roughly double the team's norm, driven by missed edge cases. The same tool has been genuinely useful for clustering 1,200 free-text survey responses. Your VP asks whether the team should stop using AI.
1. Where in the PO/PM workflow is AI-drafted content appropriate, and where must a human own the output? Anchor your answer to the roles.
Read the worked answer
Frame this by accountability, not by task type. The PO is the content authority for the team backlog; that authority is a named human responsibility and cannot be delegated to a model. The PM owns the feature set and its economics. AI sits upstream of those decisions, never in place of them.
Appropriate: first drafts of stories and acceptance criteria from a well-specified feature; synthesizing 1,200 survey responses into themes; clustering support tickets; rewriting for clarity; generating edge-case checklists to prompt the human reviewer; first-pass classification of feedback with a human audit.
Not appropriate as a final output: anything asserting a regulatory, contractual, or security obligation; anything containing customer PII; prioritization and WSJF inputs, since the model has no view of strategy, cost of delay, or stakeholder negotiation; and communications to stakeholders about commitments.
A rule you can state to the VP: AI may draft anything, may decide nothing, and may not see data you would not put on a slide in a public meeting.
The three hallucinated SOC 2 clauses are the textbook failure: the model produced plausible-sounding compliance language and nobody checked it against the obligation register. That is a PO accountability miss, not a tool miss - which is why the answer is guardrails, not a ban.
2. Design the minimum guardrails needed to keep AI assistance without repeating these two failures.
Read the worked answer
Aim the guardrails at the failures you actually had, plus the one you nearly had.
Data: approved enterprise instance only; no customer transactions or PII; anonymize or use synthetic examples. Put it in the working agreement and spot-check monthly. The two contaminated stories must be removed from the backlog, and the exposure assessed against your data-handling obligations - that is a PM-level call, not a team one.
Factual claims: any AI-drafted statement about compliance, security, or contractual terms must be traced to the source of record (obligation register, contract, policy) before it enters a story. Add it to the Definition of Ready: acceptance criteria must be verifiable and traceable.
Human gate: AI output enters the backlog only as a marked draft, reviewed by the PO, with a weekly sample audit by a second person to catch rubber-stamping.
Quality loop: the 2x defect rate is the real signal. Feed the misses back into the templates and prompts - edge cases, error paths, empty states, roles and permissions - and track defect rate on AI-drafted versus hand-written stories for the next three iterations.
The trap is banning the tool. You would lose the survey clustering, which is a genuine win, and people would keep using it off-network where you can see nothing at all.
3. How would you evaluate whether AI assistance is actually paying off, and what would you tell the VP?
Read the worked answer
Measure against the work, not the enthusiasm, and pick three metrics.
- Refinement throughput versus rework. Time to get a feature to 'ready' should fall. If review and rework hours rise by more than the drafting saves, you are net negative. Track rework hours on AI-drafted stories specifically.
- Quality. Defect escapes and missed-acceptance-criteria rate, AI-drafted versus hand-written, per iteration. The current 2x is the baseline you have to beat, not a footnote.
- Cycle time and predictability. Story cycle time and PI objective achievement, watched for the second-order effect: reviewers rubber-stamping plausible output. Sample-audit review comments to confirm the review is real.
Add one qualitative check: are POs spending the time saved on customer discovery, or simply producing more stories? More stories is not value.
Tell the VP: we are continuing with guardrails, we will know in two PIs, and here is the stop condition agreed now - for example, if defect escape rate on AI-drafted stories has not fallen to within 25% of the human baseline by the end of the next PI, we restrict AI to summarization and drafting only.
The trap is reporting 'stories generated per week', which rewards exactly the behavior that produced the fabricated compliance clauses.