Prompt and Index Versioning for a RAG Assistant in CI/CD
A RAG assistant answers questions over 120,000 internal documents. Prompt edits are made directly in the Azure portal by three engineers and one product manager. A full index rebuild takes 9 hours and runs every Sunday; an embedding-model upgrade has been deferred twice because there is no safe way to test it. There are dev, staging, and prod environments, but only prod has realistic data volume. Evaluation currently happens in production via thumbs up/down. Last month a prompt change reduced answer length and nobody noticed for six days that citation rate had dropped.
1. Design versioning and promotion for prompts, retrieval configuration, and the index.
Read the worked answer
Treat the whole answering configuration as one versioned artifact.
- Bundle, don't version in pieces. A single 'assistant config' unit: prompt templates, system message, retrieval top-k, chunking params, reranker settings, embedding model ID, and index name. Versioning the prompt alone is what let a prompt change ship against an unchanged index with no record of the pairing.
- Store in git, deploy by pipeline. Portal editing must be disabled in staging/prod (RBAC). The portal stays available in dev only.
- Index as an immutable, named artifact. Never mutate the live index in place; build
kb-2024-11-01, point config at it, cut over. This is what makes the embedding upgrade testable: build a parallel index with the new embeddings, run the eval suite against both, cut over on evidence. - Promotion = config version + index name, reviewed like code.
Gap to call out: staging lacks prod-scale data, so retrieval quality there is not predictive. Either sample a realistic subset into staging or run the eval against a prod-sized shadow index - otherwise the gate measures the wrong thing.
2. Replace the 9-hour weekly full rebuild with a strategy that keeps retrieval fresh without risking a bad cutover.
Read the worked answer
Split 'fresh' from 'consistent'.
- Incremental upsert for adds/edits/deletes into the active index, with a delete path - a rebuild-every-week pattern usually hides the fact that deleted documents never leave the index, which is a compliance exposure as much as a quality one.
- Blue/green full rebuild on a schedule you actually need (schema, chunking, or embedding change), not weekly. Build the new index alongside, run the retrieval eval suite against both, then switch the config's index name. Cutover is instant and reversible because the old index still exists.
- Keep the previous index warm for at least one cycle; deleting it immediately is what turns a rollback into a 9-hour outage.
Measure before committing: what fraction of documents change per week? If it is small, weekly full rebuilds are mostly wasted compute; if it is large, incremental upserts need more testing than a rebuild does. The right answer depends on that number, so get it first.
3. Where should evaluation gates live, given that only prod has realistic data?
Read the worked answer
Three layers, with different questions at each.
- Pre-merge (dev, small golden set): deterministic checks that need no realistic volume - citation presence, refusal behavior on out-of-scope questions, format, latency sanity. Fast, blocks obvious regressions.
- Pre-cutover (against a prod-sized shadow index, offline replay): replayed recent production questions scored for groundedness, citation rate, and answer relevance. This is where you catch the six-day citation-rate drop, because citation rate is a cheap deterministic metric - count it on every replay, do not rely on thumbs.
- Post-cutover (prod canary): a small traffic slice compares the new config's live metrics against the baseline for a fixed window, with automatic revert.
The judgement: do not try to make staging look like prod to satisfy a single gate. Instead, move the realistic evaluation to a replay against real data that runs before traffic moves. Also fix the latent problem - thumbs-only evaluation has no baseline, so nobody could have noticed the citation drop. Add the metric first, then the gate.