Streaming Assistant With Long-Context Summarization Budget
A team ships a customer-support assistant on Amazon Bedrock using a foundation model with a 200K-token context window. Each conversation turn currently sends the full transcript plus 12 retrieved policy documents (about 3,000 tokens each). Median session length has grown to 90 turns, and p95 input size now reaches 165K tokens. Symptoms: p95 first-token latency rose from 1.2s to 6.8s, throttling errors appear at peak, and monthly input-token spend tripled. The product owner wants answers to stay grounded in the retrieved policies and to preserve references to earlier turns (for example, order IDs mentioned 40 turns ago). The team must cut cost and latency without losing answer quality, and must keep responses streaming to the UI.
1. Diagnose why latency and cost grew, and propose a context-management strategy that preserves grounding and earlier-turn references while keeping streaming responses.
Read the worked answer
The growth is structural, not a model regression. Input size scales with turns because the full transcript is resent every call, and the retrieved documents are re-injected unchanged even when the new question touches only one policy. Because transformer attention and prefill cost grow with input tokens, p95 at 165K tokens explains the 5.6s latency increase and the throttling (per-minute token quotas are consumed by prefill, not output).
Strategy:
- Retrieve per turn, not per session. Run retrieval against the current user utterance plus a short rolling summary, and pass only the top 3-5 chunks after reranking. Twelve full documents is almost never necessary.
- Chunk and cite. Store policies as smaller chunks with metadata (policy_id, section). Grounding is preserved by citations, not by volume.
- Rolling summarization with a pinned facts block. Maintain a structured 'session facts' object (order IDs, product, entitlements) extracted each turn, plus a periodic summary of older turns. Send summary + facts + last N turns instead of the raw transcript. This preserves the order ID from 40 turns ago without resending 40 turns.
- Cache the static prefix. Put system prompt and stable instructions first and use prompt caching so repeated prefixes are billed and processed cheaply.
- Stream with a budget guard. Enforce a max input-token budget per call; if exceeded, summarize before calling. Streaming is unaffected because output generation is unchanged.
Validate with an offline eval set of real sessions comparing answer faithfulness and fact recall (order ID retention) before and after. Expect large input-token reductions with equal or better groundedness because irrelevant documents were also a distraction source.
2. Peak-hour throttling persists even after input tokens drop by 60%. What operational and architectural changes would you evaluate, and how would you decide between them?
Read the worked answer
A 60% token reduction that does not remove throttling suggests the bottleneck is request rate or concurrency against a per-account quota rather than raw tokens, or that traffic is bursty enough to exhaust burst capacity.
Diagnose first: separate throttling by model, by region, and by caller. Check CloudWatch metrics for Invocations, Throttles, and input/output token counts per minute, and correlate with deployment events. Determine whether one noisy client (for example, a batch job or a retry storm) dominates.
Options to evaluate:
- Retry hygiene. Exponential backoff with jitter and a capped retry count. Aggressive immediate retries amplify throttling. Ensure the client honors Retry-After style signals.
- Provisioned throughput. If baseline demand is predictable, provisioned throughput units give reserved capacity and predictable cost. Compare break-even against on-demand pricing at the observed steady-state token rate; provisioned only pays off above a utilization threshold.
- Cross-region or cross-model routing. Route overflow to a secondary region or a fallback model with a compatible prompt contract. This needs an evaluation gate so quality does not silently degrade.
- Queueing and load shaping. Put non-interactive work (summarization, batch re-embedding) on a queue with rate limits so interactive traffic keeps headroom.
- Caching. Cache identical or near-identical answers (FAQ-style) to remove duplicate calls entirely.
Decision rule: quantify each option's cost per successful request at p95 and its quality risk. Prefer retry hygiene and queueing first (cheap, no quality risk), then provisioned throughput if utilization justifies it, then routing with an eval gate. Document the chosen policy and add alarms on throttle rate so regressions are caught early.