A Model Swap, 22 Topics, and Nine Write Actions

The plan is not a plan; it is an uncontrolled experiment on live constituents.

Three specific problems:

1. A model change alters every decision surface at once. Action selection, instruction adherence, tone, and edge-case handling can all shift. With 140 actions and no trustworthy baseline, you cannot tell a model regression from an ordinary defect.
2. The suite cannot establish a baseline. 12 of 60 failing and described as flaky means the suite's signal is buried in noise. A 'flaky' test is almost always an underspecified test - it asserts on wording or on an incidental ordering rather than on behavior.
3. The audit findings are already a control failure, independent of the model. Undocumented fee waivers and a preference change on the wrong constituent are write-path defects that no model evaluation will surface, because the tests presumably check that a waiver was issued, not that it was entitled.

Before the swap:

• Rebuild the suite stratified by risk, not by topic count. The 9 write actions get exhaustive coverage; read-only FAQ topics get sampled coverage.
• Fix or delete every flaky test by replacing wording assertions with behavioral ones (was the action called; did the write payload match the entitlement check; did the agent refuse when documentation was absent).
• Establish a clean baseline run on the current model; any test that cannot pass consistently on current production behavior is a broken test, not a regression.
• Stage the swap: sandbox → full sandbox replay of the suite → a limited production cohort → full. Do not swap on Tuesday and monitor.
• Define rollback: which version pin, who calls it, how fast.

The sentence to say out loud: if we cannot state our current pass rate, we cannot detect a degradation.

Every write action gets four case classes. Read-only topics do not need this depth.

1. Happy path - asserts the action is called exactly once, with a payload containing the validated identifiers, and that the response echoes the created/updated record ID from the system rather than a described one.
2. Refusal path - the entitlement is absent. Asserts the action is not called, and that the response states the reason and the next step. For the fee waiver: no supporting document → no waiver, no exceptions. This is the case that was missing.
3. Wrong-target path - the constituent identifier is ambiguous or does not match the authenticated user. Asserts no write occurs and the agent asks rather than resolves the ambiguity. The wrong-record preference change is exactly this case, untested.
4. Failure path - the platform call errors or times out. Asserts no duplicate write on retry, and a response that says the action failed. Silent retry on a case creation is how duplicates are born.

Additional assertions I would make mechanical, not subjective:

• Payload schema check: the write payload matches the expected field set; no extra fields written.
• Idempotency check: run the same case twice; the second must not produce a second record.
• Post-condition check: query the record after the run and assert the actual state, not the agent's description of the state.

Sample size: for nine write actions, 4 classes each is 36 cases minimum, and I would add 5 variants per refusal path since phrasing varies most there. Roughly 70–80 cases. That is a small, tractable suite - the current 60 ad hoc cases are covering 140 actions, which is why it catches nothing.

Testing catches known failure shapes. These controls catch the ones you didn't imagine.

1. Split the agent's permissions by consequence. Today one agent identity holds 9 write capabilities; the fee-waiver capability is the outlier. Scope it so that waiver issuance requires a documented attachment as a server-side precondition - a validation rule or flow gate, not an instruction. If the instruction is the only thing preventing the waiver, the control is a suggestion.
2. Human-in-the-loop for the two high-consequence actions (fee waiver, complaint logging). Draft-and-confirm, not auto-execute, until six months of clean audit. The agent prepares, a staff member approves. This costs seconds and buys a real audit trail.
3. Deny-by-default action inventory. Every one of the 140 actions should have a named owner and a stated consequence level. The fact that the org cannot quickly say why the agent holds a waiver capability is itself the finding.
4. Runtime monitoring with alerting, not monthly audit. Alert on: waiver issued without attachment, write volume per topic outside 2σ of trailing baseline, and any write action on a record whose owner is not the authenticated user. Monthly audit finds three; alerting finds the fourth on the day it happens.
5. Post-swap review gate. For two weeks after the model change, sample 50 write actions against post-conditions and compare refusal rates to baseline. A shift in refusal rate - in either direction - is the earliest signal of a behavioral change.

The judgment to state plainly: the model swap is the scheduled risk; the authorization scope and the audit cadence are the chronic ones. Fixing the chronic ones also makes the swap survivable, because you finally have a baseline to compare against.