Choosing a Qwen variant for a multilingual support assistant under a latency budget
A retail team is building a customer-support assistant that must handle English, Mandarin, and Thai. They have 40,000 historical tickets with human-written resolutions, but only 3,000 are in Thai. Their product owner wants answers grounded in a 12,000-document policy corpus and wants the model to occasionally call a refund API. During a pilot on a hosted endpoint, a 72B-class dense model produced good answers but median latency was 6.5 seconds per turn, above the 3-second target. A 7B-class model met latency but hallucinated policy numbers and mixed Thai and English mid-sentence. The team has a fixed monthly inference budget and cannot fine-tune a 72B model. They can, however, run retrieval and light prompt engineering, and they can choose among dense, MoE, and smaller instruction-tuned Qwen variants.
1. Which model-selection and architecture direction should the team take, and why? Address the latency, multilingual, grounding, and tool-calling constraints together.
Read the worked answer
The team should separate capability from latency rather than picking one model for everything.
- Grounding first: The hallucinated policy numbers are a retrieval problem, not primarily a model-size problem. A retrieval-augmented pipeline over the 12,000-document corpus with citation-enforced prompting will remove most policy-number errors even on a smaller model.
- Model choice: A mid-size MoE Qwen variant is the natural compromise: it activates fewer parameters per token than a dense 72B model, so it can approach the 3-second target while retaining stronger multilingual and instruction-following behavior than a 7B dense model. If no MoE variant fits the budget, use a larger dense model only for a router/verifier role and a small model for the bulk of turns.
- Multilingual: Thai is the weak language (3,000 examples). Do not fine-tune; instead use few-shot exemplars in Thai and enforce language via the system prompt plus a post-generation language check. A model with stronger multilingual pretraining coverage matters more than raw size here.
- Tool calling: Refund API calls should be gated by a deterministic policy check (order ID, amount, eligibility) before the model is allowed to emit a tool call. The model proposes; code validates.
- Latency: Stream responses, cache frequent policy answers, and route simple FAQ turns to the small model while escalating complex multi-turn cases to the larger one.
The key judgement is that the 72B model's quality advantage was partly masking a retrieval gap, and the 7B model's failures were addressable with RAG and language controls rather than a bigger model.
2. The team proposes fine-tuning the 7B model on all 40,000 tickets to fix the Thai mixing and policy errors. Evaluate this proposal.
Read the worked answer
This is likely the wrong lever, for three reasons:
- Data imbalance: Only 3,000 of 40,000 tickets are Thai. Fine-tuning on the full set will reinforce English-dominant behavior and may worsen code-switching, not fix it. Thai examples would need heavy upweighting or curation.
- Wrong failure mode: Policy-number hallucinations come from missing grounding, not from a knowledge gap the model can learn from tickets. Tickets contain resolutions, not the authoritative policy corpus; training on them teaches the model to imitate past answers, including any past mistakes.
- Cost and risk: Fine-tuning consumes budget and creates a maintenance burden every time policies change. A RAG pipeline updates by re-indexing documents; a fine-tuned model needs retraining.
A better plan: keep the base instruction-tuned model, add RAG with citations, add Thai few-shot examples, and add a post-generation validator that rejects answers containing untranslated English tokens or uncited policy numbers. Reserve fine-tuning for style/tone if needed, using a small, balanced, human-reviewed set.