The Copilot That Re-Sends 60,000 Tokens Every Turn

Root-cause map - five symptoms, three causes, and all three proposed fixes are wrong.

• 429 bursts at 09:00 - unpaced concurrency against a fixed limit. Fix: client-side rate limiting and a queue, plus backoff that honors retry-after. Raising retries to 10 makes it worse.
• Duplicate refunds - blind retry of a request with side effects. Fix: idempotency keys; never blind-retry a non-idempotent tool call. More retries multiplies the damage.
• Half-formed JSON in stream - parsing deltas as if they were complete. Fix: buffer deltas, parse only after message_stop, or use a streaming JSON parser.
• Truncated summaries - ceiling reached. Fix: detect stop_reason == 'max_tokens' and continue or shorten. Raising the ceiling to 8,192 only moves the cliff and raises cost.
• $47k bill - 60,000 static tokens re-sent on every turn. Fix: cache the static prefix and retrieve instead of pasting. Deleting the last 20,000 tokens silently breaks answers with no measurement.

Ranking by impact: duplicates first (correctness and legal exposure, not just spend), then cost, then truncation and streaming (visible quality), then rate limiting - which is cheap and removes the retry storm that produces the duplicates in the first place.

The pattern to name explicitly: two of the three proposed fixes make symptoms quieter while increasing spend and hiding the defect.

Retries.

• Classify before retrying: retry 429, 5xx, overloaded responses, and network timeouts. Never retry 4xx request errors - they will fail identically.
• Honor retry-after when present; otherwise exponential backoff with full jitter, capped at about 5 attempts and a total wall-clock ceiling.
• Retry only when no side effect has landed. For refunds, derive a deterministic idempotency key per logical action (ticket id + action + amount) and pass it downstream so a replayed request returns the original result instead of creating a second record.
• Move anything that does not need a live answer onto a batch path; that removes the 09:00 burst for bulk work.

Streaming.

• Treat events as transport, not content: append content_block_delta text into a buffer and parse JSON only at message_stop.
• Show a typing indicator, never partially parsed fields, while streaming.
• On stop_reason == 'max_tokens', mark the response truncated and either continue or reduce scope; never store it as complete.
• Log request id, stop_reason, and token usage per call. Without those three fields you cannot debug cost or truncation at all.

Decompose before optimizing. 60,000 tokens of knowledge base on every turn means an 8-turn conversation sends roughly 480,000 input tokens. Input, not output, is the bill.

Three stacked fixes, in order of leverage:

1. Cache the static prefix. Place the system prompt and knowledge base at the start of the request, mark a cache breakpoint there, and keep everything variable (conversation turns) after it. Repeated reads of the cached prefix cost a fraction of fresh input, so turns 2–8 drop sharply. Verify with the API's cache-read counters rather than assuming it is working.
2. Stop pasting the whole knowledge base. Chunk it and retrieve the top 4–8 passages (roughly 6–10k tokens) per question. This also improves quality: 60k tokens of mostly irrelevant context dilutes attention on the passage that matters.
3. Compress history beyond six turns; keep the last three verbatim and summarize earlier ones.

Why the team's proposal is the worst option: deleting the last 20,000 tokens removes content with no measurement of which answers break, and it is irreversible at answer time. Fixes 1 and 2 should plausibly bring spend into the $15k order of magnitude on their own.

Instrument cost per conversation and cache-hit rate weekly, and re-measure after each change so you can attribute the saving.