A prompt cache with a 2% hit rate
A policy Q&A assistant assembles each request as follows: a freshly generated header containing the current date, time, and the user's account summary (name, plan, region, open ticket count); then a 6,000-token style guide; then a 3,000-token block of tool definitions; then roughly 20,000 tokens of retrieved policy excerpts; then the user's question. Traffic is about 50 requests per hour during business hours, near zero overnight. The team added cache breakpoints to the style-guide block and the tool block, expecting a large win. Measured cache-read tokens are about 2% of total input tokens and p95 latency is essentially unchanged. A junior engineer proposes adding a third breakpoint on the last retrieved excerpt 'so the answer gets cached'.
1. The breakpoints look right. Why is the hit rate 2%?
Read the worked answer
Prompt caching is an exact prefix match. A cache read only occurs if every token before the breakpoint is byte-identical to what was written. The breakpoint does not mark 'cache this block'; it marks 'everything from the start of the prompt through here is a cacheable prefix'.
The freshly generated header sits at the very start. The date and time alone change every request, so the prefix differs on every single call and every write is a miss - the style-guide and tool breakpoints are downstream of a prefix that never repeats. That fully explains 2%: the few hits are likely requests landing in the same second or a retry of an identical request.
Fix: order by volatility, not by how the model 'reads' it. Static content first - tool definitions, then the style guide - with breakpoints at the end of each. Everything volatile (date, account summary, retrieved excerpts, question) goes after the last breakpoint. A stale-feeling ordering concern is worth testing, but it cannot be worth a 98% miss rate.
Check the TTL too, but it isn't the main cause. The default 5-minute window covers a 50 req/hour business-hours pattern most of the time, but overnight gaps will re-write. The 1-hour TTL costs more to write and wins if you get several reuses - at 9,000 cacheable tokens the write premium is small next to the read discount, so it pays back after a couple of hits. Confirm with the cache-creation and cache-read token counters rather than with latency.
2. Evaluate the proposal to also put a breakpoint on the last retrieved excerpt.
Read the worked answer
It misunderstands what the cache holds, and it will mostly cost money.
Prompt caching stores token prefixes, not answers and not semantic results. There is no lookup that maps 'a similar question was asked before' to a cached response. Putting a breakpoint after the retrieved excerpts means the cached unit is 'header + style guide + tools + these 20,000 tokens of retrieval' - and those excerpts change per question, so the prefix is almost never reused. You would pay the write premium repeatedly for a hit that arrives only when the same question is asked twice within the TTL.
There is also a hard limit on breakpoints per request, so breakpoints are a scarce resource; spending one on a block that never repeats is strictly worse than spending it on static content.
What actually pays here:
- The 20,000 tokens of retrieved excerpts are uncached, full-price, every request. Reranking, deduplicating near-identical passages, and trimming to the top 8–10 chunks is the biggest lever, and it usually improves answer quality by reducing distraction.
- Retrieval results caching belongs in your application layer, keyed by normalised query - a separate mechanism from prompt caching, with its own TTL and invalidation policy when policy documents change.
- Scenario where an excerpt breakpoint does pay: a fixed document set shared across many users, where the same long preamble precedes all questions. That is not this traffic pattern.
3. How do you verify that reordering the prompt for cacheability did not degrade answers, and what do you monitor afterwards?
Read the worked answer
Reordering a prompt is a prompt change, full stop. Moving the account summary from the top to after 9,000 tokens of style guide changes what the model attends to, and position effects are real. Cacheability is a cost property; it must not be validated with cost metrics.
Verification:
- Run old and new prompt assembly over the same frozen evaluation set, paired item-by-item, and compare a bootstrap CI on the difference rather than two aggregate scores. Include a slice of questions where the account summary actually matters (plan-dependent policy answers) - that's where reordering would hurt first.
- Check abstention and refusal rates as well as correctness. A prompt reorder can quietly make the model more willing to answer from insufficient evidence.
- If the reorder does hurt, restore the position that works and cache a smaller static prefix. Correctness beats a hit rate.
Ongoing monitoring:
cache_read_input_tokensandcache_creation_input_tokensper request, as a ratio. A sudden rise in creation with flat reads means the prefix broke - usually a timestamp, a user id, or a tool-schema edit sneaking in early.- p50/p95 latency and cost per answered question, both tracked alongside quality so a cost win can't hide a quality loss.
- A small daily canary set with known answers, run through the live prompt assembly, alerting on drift.
- Log a hash of the assembled prefix alongside each response. When quality drops, the first question is 'did the prefix change?' and the hash answers it in seconds.