Improving Generative AI Output: Prompting, Grounding, and Tuning
Three levers, in escalation order
Prompting is fast, free of weight changes, and experimental. Grounding injects authoritative context at inference time. Tuning changes model weights using your examples. Always try them in that order - cheapest first.
Prompting techniques
State the task, role, and desired output format explicitly. Use few-shot examples to demonstrate format and edge cases, chain-of-thought ("reason step by step") for multi-step problems, system instructions for persistent behavior, and structured output with a schema when downstream systems must parse results. Iterate against an evaluation set rather than eyeballing single responses.
Grounding
Retrieval-augmented generation pulls passages from your corpus (Vertex AI Search, vector search) or from Google Search and places them in the prompt. This reduces hallucination, supplies freshness, and enables citations. Grounding does not modify the model.
Tuning
Supervised fine-tuning teaches style, format, and task behavior; adapter/LoRA tuning is parameter-efficient; distillation transfers a large model's behavior into a smaller, cheaper one. Tuning is weak at injecting new or frequently changing facts and does not by itself guarantee groundedness.
Traps
- Choosing tuning for "answer from policy documents that change weekly" - the answer is grounding.
- Believing RAG updates model weights.
- Treating few-shot examples as equivalent to fine-tuning.
- Assuming higher temperature improves factual accuracy.
Rule of thumb
Knowledge → grounding. Behavior, style, format → tuning. Randomness → temperature. Reasoning quality → better prompt or a stronger model.