Travel Insurer Assistant: 6.2s p95, a 5-Minute Cache, and Stale Policy Text
A travel insurer deployed a Bedrock-based assistant on its public site. It answers policy questions from a 4,000-chunk knowledge base and, for logged-in customers, is supposed to report claim status. Median latency is 1.4 seconds but p95 is 6.2 seconds against a 3-second target. Support tickets show two patterns: (a) customers were told 'your claim is approved' when a five-minute response cache held an older status; (b) the assistant answers cancellation-fee questions with confident but outdated wording from a policy superseded in March. The team proposes two fixes: fine-tune the model on the new policy PDFs, and add a 30-second cache to bring latency down.
1. Evaluate both proposed fixes and say what you would do instead for each problem.
Read the worked answer
Fix 1 - fine-tuning on the new policy PDFs: reject. Fine-tuning teaches behaviour, style, and format; it is a poor way to inject facts. It will not give citations, it must be repeated for every policy revision (four a year?), it cannot be rolled back per-document, and outdated facts baked into weights are hard to detect. The real defect is content lifecycle: the March-superseded document is still in the index. Fix the index - version each chunk with effective-from/effective-to dates, retire superseded documents, and have retrieval filter to the date of the quote. Then the base model plus RAG is correct again.
Fix 2 - a 30-second cache: reject as framed. The cache is not 'too long'; it is caching the wrong thing. Claim status is transactional, per-customer data and should never be served from a shared semantic or response cache - a five-minute or thirty-second TTL both risk showing customer A's status to customer B and both risk staleness. Status must come from a real-time read against the claims system of record via an action group, authenticated as the logged-in user.
Cache only static policy text, keyed by document version and retrieval query, where a stale entry is impossible by construction because the key changes when the policy does. That still buys most of the latency win without correctness risk.
2. The p95 latency target is 3 seconds. Describe how you would find the time, and which levers you would pull first.
Read the worked answer
Instrument before optimising. Add per-stage traces: authentication, embedding of the query, vector search, re-ranking, prompt assembly, model time-to-first-token, generation time, and any downstream tool call. Without this, the team is guessing, and p95 problems are usually one or two stages with heavy tails rather than uniform slowness.
Likely contributors and the levers, in order of cost-effectiveness
- Retrieval breadth: 4,000 chunks returning 20 per query, with re-ranking, is a large tail contributor. Retrieve more cheaply, then re-rank to top 4-5; prompt size and prefill time drop with it.
- Output length: cap max tokens and require a structured short answer. Generation dominates p95 far more than prefill.
- Model routing: a small model to classify intent and answer routine policy questions, escalating only complex or multi-part queries to the larger model. Most traffic is repetitive.
- Parallelism: fire the claims-API call (once the action group exists) concurrently with retrieval, not after generation begins.
- Streaming: does not reduce total latency but changes perceived latency substantially for a chat UI.
- Throttling: check whether the p95 tail is actually rate-limit retries or cold starts; if so, provisioned throughput or reserved capacity addresses it.
Do not reduce latency by trimming citations or skipping the live status call - those are the correctness properties you just fixed.
3. What guardrail design ensures the assistant never states a claim status it did not read from the system of record?
Read the worked answer
Make the status path structural, not prompt-based.
- Tool-only authority: the model has no claim data in its context unless an action group call returned it. Status is never inferred from conversation history, cached text, or the knowledge base.
- Typed tool response: the claims API returns a structured object - claim_id, status enum, effective timestamp, and the caller's authorisation scope. The model receives exactly this and may only select from the enum; free-text status narratives are generated from the enum, never independently.
- Response validation before display: a post-generation check confirms that any sentence containing a status term is backed by a tool result in the same turn. If not, suppress and fall back to 'check the claims portal' with a link.
- Authorisation at the tool boundary: the action group authenticates with the customer's own session, so a prompt-injected request for another customer's claim fails at the API, not at the model.
- Freshness assertion: include 'as of' the tool timestamp in the answer, so a human can spot staleness.
- Regression suite: scripted tickets including the historical 'approved vs pending' case, run on every model or prompt change.