Fine-Tuning a Chatbot on Thin Data
A mid-size retailer wants a customer-service chatbot that answers questions about returns, shipping, and product specs. They have 4,000 historical chat transcripts, of which about 600 contain a clear question–answer pair. The team proposes fine-tuning a 7B-parameter open model on the 600 pairs for three epochs, then deploying it with no retrieval component. They report that a small internal test shows 'good answers' and want to go live in two weeks. Product specs change frequently, and returns policy varies by region (US, UK, DE). The team has one GPU and no dedicated ML engineer.
1. Evaluate the proposed fine-tuning approach and identify why it is likely to underperform or fail in production.
Read the worked answer
Several problems compound:
- Data volume and quality. 600 pairs is very small for fine-tuning a 7B model; the model will likely overfit and memorize phrasing rather than learn generalizable behavior. Three epochs on 600 examples is a recipe for catastrophic forgetting of general instruction-following and for brittle, template-like answers.
- No retrieval for volatile facts. Product specs and regional return policies change frequently. Fine-tuning bakes facts into weights, which then go stale and cannot be updated without retraining. A RAG approach (retrieve current policy/spec documents, then answer) is the correct architecture for changing facts.
- Regional variation. The model must condition on region; without retrieval or explicit region context, it will blend policies and give wrong answers.
- Evaluation is weak. 'Good answers' on a small internal test is not a measurement. There is no held-out set, no slice analysis by region or question type, and no hallucination check.
- Operational constraints. One GPU and no ML engineer make fine-tuning, evaluation, and monitoring risky; a hosted API or a retrieval-only design may be more maintainable.
The proposal conflates 'customize the model' with 'make it answer correctly.' Most of the value here comes from retrieval and good prompting, not weight updates.
2. Propose an alternative architecture and a two-week plan that fits the team's constraints.
Read the worked answer
Architecture: Retrieval-augmented generation over a maintained knowledge base of product specs and region-specific policy documents, with a prompt that injects the customer's region and instructs the model to answer only from retrieved context and to say 'I don't know, let me connect you' when context is insufficient. Use a hosted model API or a small open model served behind an inference endpoint; avoid fine-tuning initially.
Two-week plan:
- Days 1–3: Assemble and clean the knowledge base (specs, US/UK/DE policies); chunk and index with metadata (region, product, effective date).
- Days 4–6: Build the RAG pipeline with region-aware retrieval and a grounded prompt; add a refusal path and citation of the source doc.
- Days 7–9: Create a 150–200 query evaluation set stratified by region and question type; measure answer correctness, groundedness (does the answer follow from retrieved context?), and refusal appropriateness.
- Days 10–12: Human review of failures, iterate on retrieval and prompt; add logging.
- Days 13–14: Limited pilot with a human handoff for low-confidence answers; define go/no-go criteria.
Defer: Fine-tuning until there is a larger, curated dataset and a clear need (e.g., tone or format), and only after RAG is stable. If fine-tuning is later pursued, use a small adapter, hold out a proper test set, and monitor for forgetting.