3% truncated JSON and a 40% input-token spike

Diagnosis. The failures are almost certainly stop_reason == "max_tokens". Output is being cut off at 512 generated tokens, and long pasted threads make the model produce preamble (summarising or echoing the thread) before it ever emits the object. The parser then sees an unterminated string and reports 'invalid JSON' - a symptom that hides the real signal, which is sitting in stop_reason.

The retry makes it worse because the message list now ends with a partially written object plus 'continue'. The model resumes mid-structure, so you can get two objects, two opening braces, or a continuation that assumes state it can't see. You are also re-sending the instruction block, the ticket, and the partial output on every retry.

Changes, cheapest first.

• Branch on stop_reason before parsing. max_tokens is a truncation event, not a parse error; alert on it separately.
• Move the static instructions into the top-level system parameter. Instructions are not conversational content, and a system block is easier to cache and to version.
• Put the ticket content last, after the instructions, and wrap it in an XML-style tag so the boundary is unambiguous.
• Instruct 'respond with the JSON object only, no prose and no code fences'.
• Prefill the assistant turn with an opening brace so the first generated token is JSON. This removes the preamble entirely and cuts median output length, which also cuts cost.
• On truncation, retry once from scratch (no conversation history) with a larger budget and a compressed ticket, then fall back to a queued human review.

It is a reasonable safety net and a poor primary fix.

max_tokens is a ceiling, not a target. You are billed for tokens actually generated, so unused headroom costs nothing in tokens. The real costs of a large global ceiling are:

1. Tail latency. A runaway generation can now run to 8,000 output tokens before it stops. At your volume that turns a 3% error rate into a 3% very slow error rate, and you have a wall-clock timeout somewhere downstream.
2. It hides the cause. The model is generating prose before JSON. Raising the cap means the failures disappear from the parser logs while the wasted output tokens stay. You lose your only loud signal.

So: raise the ceiling as insurance, but keep the diagnosis work. Concretely, keep the prefill and the no-prose instruction so median output stays small, cap the pasted thread to a fixed token budget (truncate with an explicit 'thread truncated at N tokens' marker), and treat any occurrence of stop_reason == max_tokens as an alert with a sample payload attached, not just a parse failure. Also track output token percentiles per endpoint - if p99 output is 300 and your cap is 8,000, you know the cap is only ever catching pathology.

First, measure before you optimise. Log per request: input_tokens, output_tokens, stop_reason, retry count, and ticket length decile. A 40% rise has two plausible sources and they need different fixes - a traffic-mix shift toward longer pasted threads, or a retry/looping regression. Compare tokens per ticket within length deciles; a flat per-decile number with a rising total means mix, not code.

Where the tokens are, most likely:

• Every failed request is paid for twice, and the failures are concentrated in your longest tickets - so 3% of requests can carry far more than 3% of the tokens.
• The retry re-sends instructions + ticket + partial output + 'continue', and it may loop more than once.
• The three instruction paragraphs ride along on all 12,000 requests every day.

Cuts that don't touch quality:

1. Single retry from a clean request instead of append-and-continue; cap retries at one.
2. Truncate pasted threads to a token budget using a real token count rather than an estimate, with an explicit truncation marker in the text.
3. Move static instructions to system and cache that prefix if it clears the model's documented minimum cacheable length; the read price is a fraction of the write price and pays back after a couple of reuses.
4. Drop continue-style hacks entirely; they exist to paper over a truncation bug you are now fixing.

Re-baseline tokens per successfully parsed ticket, not per request, so retries can't hide inside the average.