System Prompt Grows to 9,000 Tokens and Quality Drops
A coding assistant's system prompt has grown over 14 months from 600 to 9,000 tokens as teams appended rules for edge cases: formatting, security caveats, tone, framework-specific guidance, and 40 few-shot examples. Users now report the assistant ignores formatting rules it used to follow, and security caveats appear inconsistently. The prompt is assembled by concatenating sections from five teams with no ownership. p95 input tokens per request are 11,000 (9,000 system + 2,000 conversation). Cost per request has tripled. A new team wants to add 1,500 more tokens of database-migration guidance.
1. Explain why a longer system prompt is degrading instruction-following rather than improving it, and propose a restructuring.
Read the worked answer
More instructions is not monotonically better. Several effects are at play:
- Instruction dilution/competition: with hundreds of rules, low-priority ones (formatting, tone) get less reliable attention, especially when rules conflict or overlap. Inconsistent security caveats are a classic sign of rules competing for attention.
- Lost-in-the-middle: content buried in a 9,000-token block is less reliably attended to than content at the start or end. Rules appended over time tend to sit in the mushy middle.
- 40 few-shot examples are expensive and often redundant; many demonstrate the same pattern and crowd out the actual rules.
- No ownership means no one prunes; the prompt only accretes.
Restructure:
- Tier the prompt: a short, stable core (identity, non-negotiable safety/security rules, output contract) at the top; task-specific guidance below; examples last.
- Consolidate and dedupe rules across the five teams; resolve conflicts explicitly rather than stacking both sides.
- Move conditional guidance out of the always-on prompt into on-demand retrieval or tool calls (e.g., framework-specific or migration guidance fetched only when relevant), so the base prompt stays lean.
- Cut examples to the minimum that demonstrates distinct patterns; 40 is almost certainly far more than needed.
- Assign ownership and require a measured justification (eval delta) for additions.
2. The new team insists their 1,500 tokens of migration guidance must be in the system prompt. How do you evaluate whether to accept it, and what alternative would you offer?
Read the worked answer
Don't accept on assertion - require evidence. Run a counterfactual eval:
- Build a set of migration-related tasks plus a regression set of unrelated tasks (formatting, security, general coding).
- Measure baseline (current prompt) vs. with-migration-guidance on: task success on migration tasks, and instruction-following on the regression set (formatting compliance, security-caveat presence), plus p95 input tokens and cost.
- Accept only if migration-task gains are real and regression metrics don't degrade. Given the existing symptoms, I'd expect the 1,500 tokens to hurt unrelated tasks - that's the hypothesis to test, not assume.
Alternative to offer: make the guidance conditionally loaded rather than always-on. Options:
- A
get_migration_guidance()tool the agent calls when the task involves migrations. - Retrieval: index the guidance and inject it only when the request matches migration intent.
- A separate specialized sub-agent/prompt for migration tasks, keeping the main prompt lean.
This gives the team their guidance at full fidelity when it's relevant, at near-zero cost and zero dilution when it isn't. Frame it as "you get better results for your cases without taxing everyone else's" - which the eval should back up.