Implement Generative AI Quality Assurance and Observability
GenAI QA splits into offline evaluation (before release, against a labeled golden dataset) and online observability (after release, on real traffic). Questions often hinge on knowing which one the scenario describes.
Offline metrics you must recognize: groundedness (is the answer supported by the retrieved context?), relevance (does it address the question?), coherence and fluency (is it well-formed?), plus similarity scores against reference answers and retrieval metrics like recall@k. Groundedness is the metric for hallucination; relevance is about intent. AI-assisted evaluators are themselves model calls - they need a capable deployment and enough quota, and should be spot-checked against human labels.
Quality assurance also means red teaming and adversarial testing: prompt injection, jailbreak attempts, harmful content, PII leakage, and out-of-scope requests - with content safety filters on both prompt and completion. Build a regression suite so a prompt tweak that fixes one case does not silently break twenty others.
Observability means tracing each request end to end: prompt, retrieved documents, tool calls, model name and version, token counts (prompt/completion), latency per step, and errors - emitted to Application Insights or an equivalent store so you can compute groundedness and thumbs-down rates on production traffic and alert on drift in refusal rate or latency.
Traps. Calling a model fluent output "grounded" because it sounds confident. Measuring only latency. Deploying an evaluator that is less capable than the system under test. Logging prompts without redacting PII; forgetting that evaluation samples cost money.
Rule of thumb: if a user complains the answer was wrong, check groundedness first; if the retrieval returned nothing relevant, no prompt will save it.