Optimize Language Models for AI Applications
Choose the right technique
Start with prompt engineering, add retrieval-augmented generation (RAG) when the model needs private or fresh knowledge, and use fine-tuning when you need consistent format, tone, or task behavior. RAG never changes model weights; fine-tuning is a poor way to inject facts that change often. Expect questions that present a scenario ("policy documents update weekly") where RAG is the only correct answer.
RAG pipeline
Chunk documents with overlap, embed them with an embedding model, and index in Azure AI Search (vector, hybrid, or semantic ranking). At query time, retrieve top-k chunks, inject them into the prompt, and instruct the model to answer only from context. Track chunk size and overlap; too-large chunks dilute relevance.
Orchestration and evaluation
Prompt flow builds DAGs of LLM, Python, prompt, and retrieval nodes, with connections for credentials. Evaluation flows score outputs: groundedness (is the answer supported by context), relevance, coherence, fluency, and similarity (the only one needing ground-truth answers). GPT-based evaluators consume tokens and require a judge model deployment. Add Azure AI Content Safety - prompt shields and harm severity thresholds - as a separate control from evaluation.
Deployment and fine-tuning
Standard deployments are pay-per-token and share capacity; provisioned throughput reserves capacity for steady, latency-sensitive load. Handle 429 throttling with retry/backoff or higher quota. Fine-tuning uses JSONL chat-format files (system/user/assistant) for supervised fine-tuning, or preference pairs for DPO; always evaluate on a held-out set against the base model.
Rule of thumb
Try prompting, then RAG, then fine-tuning - in that order and cost order.