Scoping a Retrieval-Augmented Assistant for Support Agents

The failures are the classic symptoms of no grounding: a general model answers from parametric memory, which is frozen at training time and does not contain this company's policies or current pricing. It therefore confabulates plausible-sounding policy - the worst failure mode for a support tool.

The fix is Retrieval-Augmented Generation (RAG):

• Ingest and chunk the 8,000 articles, resolved tickets, and policy manual into a searchable knowledge base, with embeddings stored in a vector index.
• At query time, retrieve the most relevant passages and pass them to the model as context, instructing it to answer only from the provided context and to say "not found" otherwise.
• Cite sources so the agent can verify the answer against the original article.

This directly attacks the root cause: the model no longer relies on memory for company facts; it reads the current source of truth each time. The judgement: for company-specific facts, retrieval beats fine-tuning because content changes weekly and retrieval reflects updates immediately.

Build incremental ingestion as a first-class pipeline, not a one-time load:

• Source of truth: treat the wiki and ticket system as canonical. Run a scheduled sync (e.g., nightly) that detects changed, new, and deleted documents via timestamps or change feeds.
• Re-index changed items only: re-chunk and re-embed just the deltas; delete embeddings for removed or superseded articles so stale pricing cannot be retrieved.
• Version and timestamp chunks so the retriever can prefer the latest policy version and the model can cite an effective date.
• Staleness guard: if an answer cites content older than a threshold for a volatile topic (pricing, policy), flag it for human review.

This is the practical answer to "content changes weekly": the index must be a living mirror of the source systems, with deletion handled as carefully as insertion. A stale retrieval is as dangerous as no retrieval.

Recommend RAG first; treat fine-tuning as a later, optional refinement - and explain the trade-off:

• Fine-tuning teaches style and format, not reliable, updatable facts. It would bake in today's pricing and policies, then go stale as content changes weekly, and it cannot cite sources. For a policy-sensitive assistant, grounding + citations are worth more than tone.
• Fine-tuning also needs curated training data and evaluation effort the team lacks in 8 weeks.

Guardrails to ship at launch:

1. Answer only from retrieved context, with an explicit "I don't know / not in the knowledge base" path.
2. Always cite the article or ticket used, so agents can verify.
3. Human-in-the-loop: position it as an assistant that drafts answers for an agent, never an autonomous customer-facing bot.
4. Evaluation set of known Q&A pairs to measure groundedness and catch regressions before each weekly content update.
5. Feedback capture so wrong answers become training signal for retrieval tuning.

The judgement: match the technique to the failure mode - the failure was fabrication of facts, so the fix is grounding and verifiability, not stylistic tuning.