GitHub Certified: Agentic AI Developer study materials
GitHub Certified: Agentic AI Developer study resources

A lint-fixing agent at $4.10/run: context saturation and write-all permissions

GitHub Certified: Agentic AI Developer · Put your knowledge to work in context.

All case studies

A lint-fixing agent at $4.10/run: context saturation and write-all permissions

An agent is configured to fix lint and type errors. Its workflow grants permissions: write-all, sets max_iterations: 40, uses a 200k-token context window, and attaches the full CI log (median 180k tokens) to the first turn. Median run cost is $4.10 and median latency 14 minutes. 31% of runs exit at the iteration cap without emitting a final summary. In a sample of 60 PRs, 11 touch files outside the src/ paths the team cares about, including pyproject.toml and .github/workflows/ci.yml. Separately, the agent repeatedly re-reads the same three files after each edit, even though nothing changed between reads.

1. Explain the likely causes of the 31% iteration-cap exits and the repeated re-reads, and specify configuration changes with the directional effect you expect on cost, latency, and exit-without-summary rate.

Read the worked answer

Diagnosis: the two symptoms are one bug. 180k of a 200k window is consumed in turn one, leaving ~20k tokens for the agent's own reasoning, tool results, and diffs. Every edit then appends a diff and a new read result, so the effective window fills within a handful of turns and the early instructions are crowded out. The agent re-reads the same three files because it has lost its record that it already read them - that is a state-retention symptom, not thoroughness. max_iterations: 40 is a blunt guard that fires after the agent has already burned far more tokens than a sane run needs, and exiting without a summary means the work is unrecoverable and has to be redone.

Configuration changes:

  1. Summarize the log before attach. Parse CI output into a structured failure list (rule id, file, line, message). Lint failures compress to a few thousand tokens. Expect a large drop in per-turn cost and latency since cost scales with tokens processed per turn, not just per run - a ~90% reduction in the first-turn payload typically moves cost by several multiples, not a few percent.
  2. Externalize plan state. Have the agent write and maintain a short plan/progress file (which files read, which fixed, which remain) and re-read that each turn instead of relying on conversation memory. This addresses the re-read loop directly and gives a cheap resumption point after any exit.
  3. Replace the iteration cap with a budget plus a forced finalize turn. Reserve the last turn as finalize: no new tool calls, must emit a diff summary and remaining failures. Exit-without-summary should go to ~0%.
  4. Instrument the right signals. Watch cost/run, latency, exit-without-summary rate, and distinct-files-re-read as a proxy for state loss. If re-reads persist after (2), the plan file is not being consulted and the prompt scaffolding is wrong - don't just raise the iteration cap.

Do not simply buy a bigger window. It raises cost and defers the state-management problem without solving it.

2. Eleven of sixty PRs touch files outside the intended scope. Evaluate `permissions: write-all` plus prompt instructions telling the agent to stay in `src/`, and design the replacement.

Read the worked answer

Verdict: the prompt instruction is not a control. It is a request. The agent's ability to write .github/workflows/ci.yml comes from the token, not from the instructions. When the agent sees a workflow file that is failing, editing it is a locally reasonable action, and a suggestion in the system prompt competes with that reasoning under a saturated context window. The 11/60 = 18% out-of-scope rate is what a non-enforced boundary looks like.

Replacement design - enforce at three layers:

  1. Token scope. Drop write-all. Grant contents: write on the branch only, plus the minimum read scopes needed to see checks and PRs. Do not grant workflows, actions, packages, or administration - workflow files cannot be modified by a token lacking workflows permission, which makes .github/workflows/ci.yml unwritable by construction.
  2. Path allow-list at the tool layer. Wrap the file-write tool so that any path not under src/ is rejected before the write executes, and the rejection is returned to the agent as a structured error. Deny by default; anything on an explicit deny-list (**/ci*.yml, pyproject.toml, lockfiles, CODEOWNERS) is a hard stop. This is deterministic and auditable.
  3. Branch and output shape. Have the agent work on a branch and open a PR rather than push to a protected branch, so a human merge is the final gate for anything surprising. Add a diff-scope check in CI: if the PR touches paths outside src/, fail the check and label it for review.

Why all three and not just one. Layer 1 stops the category of catastrophic edits; layer 2 stops in-scope-looking mistakes and produces a clean signal you can count (scope-rejection rate); layer 3 catches anything that slipped past both and makes the human accountable for the residual.