Troubleshooting Assistant: RAG, Fine-Tuning, and Citation Reliability
TelcoNow wants a troubleshooting assistant for 9 customer intents, including SIM activation, roaming, and voicemail. It has 3,200 human-labeled dialogues, 14,000 knowledge-base articles updated daily, and 900 historical cases with correct resolutions. The current prompt-only Gemini prototype reaches 78% intent accuracy and 64% answer correctness on a 600-case holdout; 11% of answers cite a policy that does not exist. The business target is 92% intent accuracy and 90% answer correctness, with p95 latency under 2.5 s. Engineers propose supervised fine-tuning on all 3,200 dialogues because 'the model needs to learn the domain'. The knowledge base changes weekly, and legal requires every answer to cite an approved article. Only 40% of the labeled dialogues include a verified resolution. The team has six weeks and a limited evaluation budget.
1. Choose among prompt engineering, retrieval-augmented generation, and supervised fine-tuning. Justify the choice and describe the role of each technique.
Read the worked answer
RAG is mandatory; fine-tuning the answer generator is not the first move. Legal requires citations to approved articles, and the knowledge base changes weekly. A fine-tuned model bakes policy into weights and cannot reliably cite the current article; it would also learn from 60% of dialogues without verified resolutions. The 11% fabricated citations are a retrieval and grounding failure, not a style failure.
Use RAG over a versioned, approved-article index. Add prompt engineering for answer format, abstention, and citation rules. Fine-tune a smaller intent classifier or router on the 3,200 dialogues if intent accuracy remains below target; that task benefits from labeled examples and does not need dynamic policy knowledge. If answer style still needs work, fine-tune only on verified resolutions after RAG is in place.
Enforce citation at generation time: the model must return an article ID from the retrieved set, and a post-processor rejects answers whose citations are not in that set. This directly addresses the hallucinated-policy failure and keeps the system current as articles change.
2. Design the offline evaluation and holdout strategy. Include how you would measure citation correctness without exhausting the evaluation budget.
Read the worked answer
Split by time and by customer, not randomly, and deduplicate near-identical dialogues. Hold out the most recent period so the evaluation reflects current articles and intents. Stratify the 600-case holdout across all 9 intents, with extra cases for roaming and voicemail if they are high-volume. Keep a small adversarial set of withdrawn or superseded policies to test abstention and citation validity.
Measure retrieval recall@k, intent accuracy, answer correctness, citation precision, and citation entailment. For citation correctness, first run an automated check that every cited ID exists in the retrieved set and is currently approved. Then use a calibrated LLM judge on a sample, with human review of disagreements. Do not rely on the judge alone for legal citations.
To control cost, use automated checks on all cases and human review on a stratified 15–20% sample plus all failures. Report confidence intervals; 600 cases is small for 9 intents. Include p95 latency and abstention rate, because a model that answers less often but correctly may beat one that guesses.
3. If the team still decides to fine-tune, what data selection and safeguards would you require?
Read the worked answer
Fine-tune only on dialogues with verified resolutions, balanced across the 9 intents, and exclude unverified or contradictory cases. Include examples of abstention and escalation so the model does not learn to always answer. Remove PHI and customer identifiers, and hold out by customer and time.
Prefer parameter-efficient tuning and keep the base model plus adapter versioned in Vertex AI Model Registry. Evaluate against the RAG baseline, not only against the old prompt. Check for catastrophic forgetting on general troubleshooting intents and on the adversarial withdrawn-policy set. Require citation validity to remain at or above the RAG baseline.
Deploy behind a canary with a circuit breaker. If citation validity or answer correctness drops, roll back to the RAG-only path. Fine-tuning can improve intent routing and tone, but it should not be the mechanism for keeping policy current or for guaranteeing citations.