Context Management, Retrieval, and Prompt Caching
Manage the window deliberately
Every request resends the full conversation, so token cost grows superlinearly across turns. Budget context explicitly: system prompt, tool definitions, retrieved documents, history, and room for the reply. Retrieve only relevant chunks, and summarize or drop old turns once they stop informing the current step.
Prompt caching
Caching is prefix-based. You mark a breakpoint with cache_control: {"type": "ephemeral"} on a content block, and everything up to and including that point becomes cacheable if it matches the previous request exactly. Cache writes cost a premium, reads are heavily discounted, and entries expire on the order of minutes, refreshed on each hit - so caching pays off for repeated large preambles in short bursts, not one-off calls.
The classic failure is ordering: put static content first (system prompt, tool definitions, long reference docs) and variable content last (the user's question). If anything before a breakpoint changes, everything from that point forward is a miss. A timestamp at the top of the system prompt silently destroys your cache.
Retrieval quality
Chunk on semantic boundaries with overlap, and prefer hybrid keyword-plus-embedding search with a reranking pass over pure vector similarity. Evaluate retrieval separately from generation - an answer that is wrong because context was missing looks identical to one that is wrong because of reasoning.
Trap to watch
Assuming cache hits are guaranteed. They are best-effort; code must be correct at a 0% hit rate, treating caching purely as a cost and latency optimization.