Optimize Generative AI Systems and Model Performance
Optimization questions ask you to trade quality against cost and latency without breaking grounding. Work the levers in order: prompt, retrieval, model, infrastructure.
Prompt-level: shorten and restructure the system message, put instructions at the start and the query at the end, request concise output, use structured output to cut parsing, and constrain the context window to only what retrieval justifies. Retrieval-level: tune chunk size and overlap, use hybrid search (keyword + vector) with semantic reranking, set top-k deliberately (too high floods the context and reduces groundedness while raising tokens), and consider query rewriting for weak user queries.
Model-level: choose the smallest model that passes your evaluation suite, use model routing (cheap model first, escalate on low confidence or failed validation), or distill/fine-tune when the goal is behavior - tone, format, task adherence - not fresh facts. RAG is for knowledge; fine-tuning is for form. Infrastructure-level: streaming for perceived latency, caching identical or semantically similar requests, provisioned throughput for steady load, batch/offline processing for non-interactive work, and multi-region deployments with graceful failover.
Measure before and after with the same evaluation suite and track cost per successful answer, not tokens per request - a cheaper model that fails and retries is more expensive.
Traps. Answering "improve accuracy" with "increase top-k." Fine-tuning to inject frequently changing facts. Adding a semantic cache where responses must be personalized or time-sensitive. Optimizing latency while ignoring that the evaluator model dominates spend.
Worked example: a chat app is slow and expensive. Enable streaming (perceived latency), cap top-k at 5 with hybrid + reranker (fewer tokens, better grounding), route simple intents to a small model, and cache FAQs - then re-run the evaluation suite to confirm groundedness did not regress.