Implement generative AI solutions
Prompt structure
A chat completion request is a list of role-tagged messages: system (behaviour and constraints), user, and assistant (few-shot examples or prior turns). The system message is the cheapest place to fix tone, format, and refusal behaviour. Sampling parameters - temperature, top_p, max_tokens, stop sequences, seed - control variability and length, not knowledge.
Grounding (RAG)
Chunk source documents, embed them, store them in an Azure AI Search index, retrieve the top matches at query time, and place them in the prompt with instructions to answer only from the provided context and cite sources. Hybrid retrieval (keyword + vector) with semantic ranking generally beats pure vector search. Instruct the model to say it does not know rather than invent.
Output contract
When downstream code must parse the answer, use structured outputs or JSON mode plus a schema, rather than asking politely for JSON in prose. Function/tool calling is for actions (look something up, create a ticket); retrieval is for knowledge. Confusing the two is the most common design error.
Quality and safety
Evaluate with groundedness, relevance, coherence, and fluency metrics over a test set; regression-test prompts when you change models. Apply content filters and prompt shields, and log prompts and responses to Application Insights for review.
Worked rule: if an answer is fluent but factually wrong, inspect retrieval (chunking, index freshness, top-k) before touching temperature - temperature zero does not make an ungrounded model correct.