Cost and Quality Trade-offs in a High-Volume Summarization Service
A media company summarizes 500,000 articles per month. Each article averages 4,000 tokens; summaries average 300 tokens. The service uses a large foundation model on-demand, costs about $0.03 per article, and takes 9 seconds at p95. Product wants summaries in under 3 seconds and a 50% cost reduction. A smaller model is 4x cheaper and 3x faster but scored 12 points lower on an internal quality rubric (0-100 scale) in a small test. The team also suspects 15% of articles are near-duplicates and that many summaries are never read. They need a plan that meets the targets without unacceptable quality loss.
1. Propose a cost and latency optimization plan, and explain how you would validate that quality remains acceptable.
Read the worked answer
The targets require attacking volume, model choice, and output length separately, and validating quality with a proper eval rather than a 12-point anecdote.
1. Reduce volume. Detect near-duplicate articles (MinHash or embedding similarity) and reuse or adapt existing summaries. If 15% are duplicates, that is a direct 15% cost cut with no quality loss. Also determine which summaries are actually consumed; for low-traffic categories, consider generating summaries lazily on first request or batching them off-peak.
2. Tier the model by difficulty. Route articles by length, complexity, or category. Use the smaller model for straightforward articles and escalate to the large model for long or complex ones, or when a confidence/quality check fails. A cascade with a quality gate often captures most of the savings while protecting quality on hard cases.
3. Control output length. Enforce a max summary length and a target structure (for example 3-5 bullet points). Output tokens are a large share of cost; trimming from 300 to 200 tokens saves roughly a third of output spend.
4. Prompt and caching. Put stable instructions first and use prompt caching. Cache summaries for repeated URLs.
5. Batch where latency allows. Non-interactive summaries can use batch inference at lower cost; reserve real-time for user-facing requests.
Validation: Build a stratified eval set of at least 200 articles spanning categories and lengths, with a rubric covering factual coverage, no hallucinations, and readability. Have humans score a subset and calibrate an LLM judge against them. Compare large-model baseline, small-model, and cascade on the same set with confidence intervals. Set a quality floor (for example, no more than 3 points below baseline and zero increase in hallucination rate). Only ship configurations that pass. Measure real p95 latency and cost per article in a canary before full rollout.
2. Design the monitoring and governance needed to keep cost and quality under control after launch.
Read the worked answer
Launch is the start of the risk, so build observability and guardrails around both dimensions.
Cost monitoring. Track cost per article, per model, per category, and total monthly spend. Break cost into input tokens, output tokens, and any retrieval or tool calls. Set budgets and alarms at 80% and 100% of target, with anomaly detection on daily spend. Attribute cost to teams and features via tagging so chargeback is possible. Watch for cost regressions after prompt or routing changes.
Quality monitoring. Continuously sample production summaries (for example 1-2%) and score them with the calibrated judge plus periodic human review. Track hallucination rate, coverage, and length compliance. Alert on drift in score distributions. Include a feedback channel for editors to flag bad summaries; triage and feed confirmed failures into the eval set.
Routing and fallback health. Monitor the cascade: share of articles sent to the large model, gate failure rate, and fallback latency. If the small model's gate failure rate rises, cost and latency targets silently break; alarm on it.
Governance. Version prompts, routing rules, model versions, and eval sets together. Require an eval pass and canary before changing any of them. Document the quality floor and the approval path for exceptions. Review monthly: cost trend, quality trend, and incident log.
Incident readiness. Define rollback to the previous configuration and a kill switch to the large model if quality or cost breaches thresholds. Run a game day to confirm the team can execute rollback quickly. This keeps the 50% cost and 3-second targets durable rather than a one-time win.