Three chained agents, shared token, and a stale-approval merge

Why stale approvals merged. The merge agent's condition was evaluated against state, not against a bound artifact: "checks green + approval exists." An approval is attached to a commit SHA at the time it is submitted, but the agent never compared that SHA to the current head. Any push after approval - including a push by the fixer agent - leaves the approval valid in the UI unless the repo is configured to dismiss it, and the agent happily merged approval-for-A onto commit B.

Why the races happened. Reviewer and fixer are both triggered by events on the same PR with no mutual exclusion. There is no ordering guarantee between pull_request.opened and pull_request_review_comment, and both call the same push endpoint. Two writers, one branch, no lock.

Ordering and concurrency model:

1. Single writer per PR. Give the fixer agent exclusive write ownership of the branch. The reviewer is read-only on code - it produces comments and verdicts, never commits. That removes the race by construction rather than by scheduling.
2. Serialize on the PR, not on the agent. Take a per-PR lock (a concurrency group keyed by PR number, or a durable lease stored on the PR) around any write. On lock contention, queue, don't retry blindly - a blind retry is how you get more force-push races and more 403s.
3. Bind the merge to an artifact. The merge agent must merge exactly (head_sha, review_id) where the review's commit_id == head_sha and the review state is APPROVED. If a push happens, the SHA changes and the merge condition is re-evaluated as unsatisfied. Enable the repository's dismiss stale approvals on push setting as defence in depth - the agent should not be the only thing enforcing this.
4. Use the merge queue. A merge queue serializes merges against current main, so a PR that is green-but-behind gets retested rather than merging on a stale base.
5. Record the decision. Emit a merge audit record with PR number, head SHA, review ID, check-run IDs, and the actor. That record is what resolves the "which review did it act on" question during an incident - and it is required for the accountability story in any audit.

Read the failure correctly. Secondary rate limits are abuse detection, not the primary hourly budget - they respond to burstiness and concurrency, so retrying harder is the worst possible response and is probably part of why 8% fail. Meanwhile a 4x spike on a shared 5,000/hr budget across three agents means no agent can reason about its own headroom; they contend invisibly.

What I would stop first: retrying on 403 without jitter. Immediate retries amplify the burst that triggered the limit, converting a short stall into an outage window. Replace with exponential backoff plus full jitter, capped, and treat a secondary-limit response as a signal to shed load for that PR rather than to try again.

Backpressure design:

1. Split budgets. Separate tokens (or at least separate budget accounting and concurrency caps) per agent role, so the reviewer's latency is not destroyed by a fixer burst. Cap concurrency per agent explicitly.
2. Prioritize by consequence. The merge agent's work is on the critical path to shipping; the reviewer's can wait minutes. Give merge > fixer > reviewer priority in the queue, and shed reviewer work first under pressure. Queued, not dropped - review latency of a few minutes is invisible; a stalled merge queue is not.
3. Admission control at the event layer. Cap in-flight runs globally and per PR; coalesce duplicate events (multiple synchronize events on one PR in a burst should collapse to one run keyed by head SHA). Idempotency key = (pr, head_sha, agent_role) so a re-delivered event is a no-op.
4. Smooth the spike deliberately. Spread scheduled work away from 09:00–11:00 UTC and add initial jitter to any batch trigger. Many 4x spikes are self-inflicted synchronization, not user demand.
5. Make saturation visible. Emit queue depth, wait time, and remaining budget per role. If you cannot see queue depth, you cannot tell backpressure from a hang, and the on-call response will be to restart things - which makes it worse.