The acceptance criteria that stopped surviving refinement

Separate the symptom from the cause. The symptom is a falling acceptance rate plus fabricated references. Three candidate causes fit the evidence, and they are not equally likely:

1. Context degradation. One ever-growing thread per product area means the model is fed a mixture of current and stale material, and long conversations get summarised or truncated internally. The team's 'it remembers context' belief is the tell: relevance is being assumed rather than engineered.
2. No grounding in a source of truth. Nothing in the workflow points the model at the actual regulation, the actual glossary, or the actual system behaviour. When the source is absent, a generative model produces plausible text rather than an admission of ignorance - hence 'clause 4.7.2'.
3. Prompt and instruction drift. Rough notes in, structure demanded out, no stated format or citation requirement, so the output varies with whoever pasted what.

Why fine-tuning is the wrong first lever. Fine-tuning shapes style and behaviour; it does not give the model reliable knowledge of your regulation or your codebase, and it will not stop fabricated citations. It also needs a curated, labelled dataset - eighteen months of backlog contains the team's own errors and stale decisions, so you would be training on the mistakes you are trying to remove. It requires an evaluation set to prove it helped, and ongoing maintenance when the domain changes. For a team with no ML capability on a two-week cadence, that is a large, slow, unprovable bet against a cheaper hypothesis.

The reasoning to model for the PO: prefer the cheapest intervention that can discriminate between the hypotheses. Fixing context and grounding is a change to how the team works, testable inside one Sprint. Fine-tuning is a change to a vendor artefact, testable only after weeks. Try the cheap one first and let the data decide.

Experiment A - fresh, grounded context per item. Each story gets a new thread. The PO's notes are pasted together with the specific source extracts that matter (the actual regulation section, the glossary entry, the relevant API contract). The prompt requires the model to cite which extract each acceptance criterion came from.

Measure: proportion of acceptance criteria surviving refinement unedited (baseline 30%), and the count of citations that do not resolve to a real source (baseline: two known fabrications).

Experiment B - verification step in the workflow. Any AI-drafted criterion with a citation is checked by the author against the source before refinement. This is not a new ceremony; it replaces part of the existing refinement preparation.

Measure: rework after Sprint Review - criteria that had to be changed because the behaviour or the rule was misdescribed.

Pre-agreed decision rule. Because the sample is small, do not declare victory on one Sprint's headline number. Agree in advance: if unedited acceptance rises meaningfully above baseline and unresolvable citations fall to zero, keep the practice and stop talking about fine-tuning. If acceptance is flat but fabrications disappear, keep B (it is a quality gain) and investigate A further. If neither moves, the hypothesis was wrong - escalate to whether the notes themselves are too thin, before reaching for any model change.

The reasoning to show the team: an experiment is only useful if it is falsifiable and if you decided the kill criterion before you saw the result. Writing the decision rule first is what stops the team retro-fitting a story to whatever number appears.

Reframe: the experiments are not extra work competing with the Sprint Goal - they are how the Sprint Goal gets met. Right now the team is absorbing rework from bad acceptance criteria in every refinement and every review. That cost is already being paid; it is just invisible because it is spread across the Sprint rather than booked to a line item.

Three concrete moves:

• Fold A into refinement, not around it. The team is drafting acceptance criteria anyway. Doing it in a fresh, grounded thread is the same activity with a different input, not a new meeting.
• Make B a Definition of Done item, not a project. 'Citations verified against source before refinement' is one line in the DoD. The DoD is how a team makes quality non-optional without adding ceremony - that is exactly what it is for.
• Make the cost visible. Track rework-after-review for two Sprints alongside the acceptance rate. If the numbers hold, the team will argue for the change itself; if they do not, you have saved the team from a change that was not worth making.

What to avoid: defending the experiment as a Scrum Master's initiative that the team owes you. The moment it feels like a compliance activity, the evidence stops being trustworthy. Offer it as the smallest thing that could settle an argument the team is already having every refinement.