Five Runs, Five Different Answers

Temperature 0 picks the highest-probability token at each step, but it does not guarantee identical output. Serving stacks batch requests dynamically and floating-point reduction order in batched matrix operations varies with batch composition, so near-tied logits can flip; GPU kernels and model revisions add further variation. Determinism is therefore not achievable by tuning, and fine-tuning would not change it. Engineer for variability instead: use the provider's structured or JSON output mode rather than trusting the words in the prompt; define a schema and validate every response; on validation failure retry once with the validation error appended, then fall back to a deterministic rule-based path. Constrain the output space, since severity is an enum and anything outside the allowed values should be rejected. Cut max_tokens to what the schema needs, because the three-times-longer response is wasted tokens and a truncation risk. Track the valid-JSON rate as a metric rather than assuming it.

Token counts are not character counts, and the 400 means the assembled prompt exceeded the context window: runbook plus log lines plus stack trace plus system prompt and schema. Two errors compound: no token budget and no prioritisation. Fix by budgeting explicitly, allocating tokens per section (system, runbook, logs, output), estimating with the model's tokenizer rather than dividing characters by four, and trimming the lowest-value section first. A full stack trace is rarely more useful than its first frames and the exception message; extract those, keep the last N log lines, and retrieve only the runbook section relevant to the detected service instead of pasting 20 KB. Put the instruction and schema at the end so they are not the part that gets truncated. If the input is genuinely large, use map-reduce: summarise log windows separately, then reason over the summaries. Add a pre-flight size check so oversized prompts are trimmed with a clear error instead of an opaque 400.

Streaming straight into the ticket publishes partial, unvalidated text: the truncated object becomes the permanent record of the incident, and a severity flip is visible to responders mid-generation. Buffer instead. Stream to the UI for perceived responsiveness if desired, but keep the authoritative write behind validation: accumulate the full response, check finish_reason for a length stop, which means the object is incomplete and must not be accepted, parse against the schema, then persist. Add a stop sequence that terminates the object cleanly and set max_tokens to the schema's realistic maximum. Store the validated fields plus the raw response in a debug field, and record provenance (model, prompt version) so a later disagreement can be investigated. Where the schema check fails or confidence is low, route to a human rather than guessing, because a wrong P1 severity costs far more than manual triage. Treat the endpoint as an unreliable dependency: timeouts, bounded retries with jitter, and a queue so a slow model never blocks incident creation.