LLM Support Assistant: Explain the Regression Before Scoring It
A financial-services company runs an LLM support assistant on Kubernetes. The assistant uses a hosted model plus an internal retrieval service over 1.2M policy documents. Over three days, p95 end-to-end latency rose from 2.1 s to 6.8 s, cost per conversation rose 38%, and thumbs-down feedback doubled. The model provider reports normal status. Traces exist for the API gateway and retrieval service, but LLM calls are instrumented only with total duration; prompt version, token counts, retrieval document IDs, and finish reason are missing. The team can deploy one change this sprint: either add OpenTelemetry GenAI semantic-convention spans around model calls and retrieval, or enable a sampled evaluation pipeline that scores 2% of conversations for groundedness. Product wants the fastest path to explain the regression and avoid repeating it.
1. Which single change should the team deploy first, and what evidence would it produce? Justify the choice against the alternative.
Read the worked answer
Deploy the instrumentation first. The immediate problem is attribution, not measurement of quality. The team already knows latency, cost, and satisfaction moved; it cannot tell whether the change came from a prompt revision, more retrieved context, a slower retrieval call, longer model output, or retries.
A GenAI span per model call should carry at minimum: model name and version, prompt template version, input and output token counts, finish reason, retrieval call latency, number of retrieved documents, and retrieval document IDs. A parent span should cover the whole conversation turn. Those attributes let the team break p95 latency and cost down by prompt version, model, retrieval latency, and token count, and then join bad outcomes to the exact retrieved passages.
The evaluation pipeline is useful but is the weaker first move here: it samples only 2% of traffic, produces a lagging quality score, and does not explain a 6.8 s p95 or a 38% cost increase. It also cannot separate retrieval degradation from generation degradation. Deploy instrumentation now, then use it to target evaluation where it matters. Use stable custom attribute names as well as emerging semantic conventions, because conventions are still evolving.
2. Form three hypotheses for the simultaneous latency, cost, and quality regression, and describe the exact telemetry that would distinguish them.
Read the worked answer
Prompt or context inflation. A prompt revision increased instructions or top-k retrieval, so each call sends more input tokens and the model produces longer answers. Distinguish with input-token counts, retrieved-document counts, prompt version, output-token counts, and finish reason. Expect input tokens and retrieval documents to rise together, with model time rising but retrieval latency roughly unchanged.
Retrieval degradation. An index update or vector-store change made retrieval slower or returned poorer passages. Distinguish with retrieval span latency, document count, document IDs, and a groundedness score joined to those IDs. Expect retrieval latency to dominate the end-to-end increase and quality to drop even when prompt version and token counts are stable.
Model behavior or retry amplification. The provider is returning longer outputs, more retries, or a fallback model. Distinguish with model name, output tokens per request, finish reason, retry count, and per-attempt latency. Provider status pages are not sufficient evidence; per-call attributes are.
Cross-tabulate latency and cost by prompt version and retrieval latency bucket. If one prompt version shows both higher token counts and worse thumbs-down rates, the prompt is the likely cause. If retrieval latency rises only for certain document collections, investigate the index.
3. Define a small SLI and alert set for this assistant that catches regressions without alerting on every bad answer.
Read the worked answer
Use a small number of user-centered SLIs, all sliced by prompt version and model:
- Latency: p95 end-to-end turn latency and p95 retrieval latency, with a target such as 3 s.
- Cost: tokens per conversation and cost per successful conversation, compared with a seven-day baseline.
- Quality: groundedness evaluation score on a sampled set, thumbs-down rate normalized by conversation volume, and refusal/error rate. Track sample size so small samples do not trigger alerts.
- Availability: failed turns by finish reason and provider error code.
Alert on multi-window burn rates for latency and error SLOs, not on single slow requests. Alert on cost anomalies only when the increase persists across two windows and exceeds a meaningful percentage. For quality, require a sustained drop over at least two evaluation windows with a minimum sample count, and route the alert with the affected prompt version and example conversation IDs. Avoid paging on individual thumbs-down events; use them for triage and weekly review. This keeps the team responsive to systemic regressions without turning normal variation into noise.