Fine-Tuning a Classifier When the Label Set Keeps Moving
A support organization wants to auto-route incoming tickets into 14 categories using a fine-tuned model. They have 22,000 historical tickets labeled by agents, but labels are noisy (agents disagree), the category taxonomy changed twice in the past year, and 3 categories were added last quarter with only 40 examples each. A baseline prompt-only classifier on a foundation model reaches 78% accuracy overall but only 31% on the new categories. Leadership wants a fine-tuned model because 'fine-tuning is more accurate.' The team has a two-week window and a modest budget.
1. Evaluate whether fine-tuning is the right first move, and design an approach that handles noisy labels, taxonomy drift, and rare new categories.
Read the worked answer
Fine-tuning is not automatically more accurate; it is most valuable when you have consistent labels, a stable taxonomy, and a need for lower cost or latency at high volume. Here the label noise, taxonomy churn, and 40-example classes are the dominant problems, and fine-tuning on noisy, drifting labels would bake in the errors.
Step 1: Establish a trustworthy baseline and eval set. Build a stratified test set (at least 30 examples per category, more for common ones) with labels adjudicated by a small panel. Measure the prompt-only baseline on it. The 78% figure may itself be inflated by noisy labels, so re-measure against adjudicated labels before concluding fine-tuning is needed.
Step 2: Reduce label noise. Use agreement statistics to find low-confidence tickets. For disagreements, use a stronger model or human adjudication on a sample. Consider training only on high-agreement examples plus a small adjudicated set. Optionally use a model-assisted labeling pass with human review for the new categories.
Step 3: Handle rare categories. With 40 examples, fine-tuning a full model will overfit. Options: (a) use retrieval-augmented few-shot examples at inference for rare classes; (b) use a hierarchical approach where a coarse model routes to a group and a second step disambiguates; (c) collect more examples via active learning, prioritizing tickets near the decision boundary; (d) use parameter-efficient fine-tuning (LoRA-style adapters) with strong regularization and class weighting, and evaluate per-class recall, not just accuracy.
Step 4: Decide with data. Compare on the adjudicated test set: prompt-only, few-shot with retrieval, and fine-tuned adapter. Track macro-F1 and per-class recall, especially for the new categories. Fine-tune only if it wins on macro-F1 and cost at the expected volume.
Step 5: Plan for taxonomy drift. Keep the label set as configuration, version the training data with taxonomy version, and design a re-labeling and retraining cadence. A category change should trigger a data audit, not a silent retrain.
2. Design the continuous evaluation and retraining process so that accuracy does not silently degrade as ticket language and categories evolve.
Read the worked answer
Treat the classifier as a monitored system with explicit quality gates.
Data and versioning. Version training sets, label taxonomy, prompts, and model adapters together. Every trained model records which data snapshot and taxonomy version it used. When the taxonomy changes, mark affected models as stale.
Evaluation harness. Maintain a fixed, adjudicated golden set plus a rolling 'recent' set sampled from the last 30 days. Report macro-F1, per-class precision/recall, and a confusion matrix. Per-class metrics matter because a rare category can collapse while overall accuracy stays flat. Also track calibration (confidence vs. correctness) so low-confidence routing to humans works.
Production monitoring. Log predicted class, confidence, and (where available) agent corrections. Track the correction rate per class and the share of tickets routed to 'other' or human review. Alert when correction rate for a class rises beyond a threshold or when the input distribution shifts (new product names, new vocabulary). Use embedding-based drift detection on incoming tickets to catch vocabulary change early.
Human-in-the-loop feedback. Agent re-categorizations are the cheapest source of fresh labels. Feed corrected tickets into a review queue, and periodically promote a curated batch into the training set. This creates a continuous labeling loop rather than a one-time dataset.
Retraining cadence and gates. Retrain on a schedule (for example monthly) and on triggers (taxonomy change, drift alarm, correction-rate breach). A new model is promoted only if it beats the current model on the golden set and does not regress any class below a floor. Use shadow deployment or a small canary to compare live behavior before full rollout, with automatic rollback on regression.
Cost control. Because retraining is frequent, prefer parameter-efficient adapters and reuse the base model. Cache evaluation runs and only re-run full suites on material changes.