Evaluation, Monitoring, and Optimization of Context Pipelines
This domain tests how you prove a context pipeline works and improve it without guesswork. The exam covers offline evaluation sets, retrieval and answer metrics, LLM judges, production tracing, drift monitoring, and controlled optimization.
Evaluation design
- Build a golden set with questions, expected source documents, expected answers, and unanswerable cases. Include hard negatives and permission-boundary cases.
- Measure retrieval separately from generation: recall@k, mean reciprocal rank, and nDCG for retrieval; correctness, groundedness, citation accuracy, and refusal quality for answers.
- Use LLM judges only after calibrating them against human labels. Prefer deterministic checks for citations, schema validity, and forbidden content.
- Track operational metrics: latency, token usage, embedding cost, tool error rate, and index freshness.
Optimization and monitoring
- Change one variable at a time: chunk size, overlap, embedding model, top-k, reranker, filters, or prompt structure.
- Monitor production traces for retrieval misses, tool failures, stale evidence, and drift in user questions.
- Feed production failures back into the golden set and re-run evaluation before changing the index or model.
Worked example
A team compares 500-token and 800-token chunks on a 120-question evaluation set. Both use the same embedding model and top-k 10. The 500-token version has higher recall@10 but lower citation accuracy because tables are split; the 800-token version keeps tables intact and improves groundedness, but costs more tokens. They add a table-aware chunking rule, re-run the set, and confirm recall and groundedness improve without exceeding the latency budget. Reasoning: chunking affects both retrieval and citation quality; changing chunk size and table handling separately would obscure which change helped.
Traps
- Evaluating only the final answer and never measuring retrieval recall.
- Using a golden set that contains no unanswerable or permission-restricted questions.
- Trusting an LLM judge without calibration against human labels.
- Optimizing cost or latency while ignoring groundedness and citation accuracy.
- Lacking production traces, so failures cannot be reproduced or added to evaluation.