Context, Token, and Output Management
Budgeting the window
Every request has a context window shared by input tokens and generated output. Tokenization is not word-count: punctuation, code, and non-English text often consume more tokens per word. Estimate with a tokenizer, then leave headroom. Set max_tokens deliberately; too low truncates answers, too high permits runaway cost. Prompt caching can reduce cost and latency for repeated long prefixes, but it does not change model behavior or guarantee identical answers.
Managing long inputs
For long documents, place the instruction first, the document next, and the question last; this sandwich helps the model attend to both. Chunk and summarize when content exceeds the window, and retrieve only relevant passages rather than stuffing everything. Watch for "lost in the middle": models may attend less reliably to facts buried in the center of very long contexts.
Traps
A frequent error is assuming the context window equals the maximum output length. Another is ignoring output tokens when budgeting: input plus max_tokens must fit. Candidates also assume truncation is harmless; cutting the middle of a document can remove the exact fact needed. Do not assume caching makes responses deterministic.
Rule of thumb
Budget: input tokens + max_tokens + safety margin ≤ context window. For a 200k-token window, a 10k-token prompt, and a 2k-token answer, you are fine; but a 195k-token prompt with max_tokens 8k will fail or truncate. Measure, don't guess.