LoRA Fine-Tune for Schema-Constrained Function Calling
A team is adapting a 13B decoder-only model to emit strict JSON for a payment API. The corpus has 38,000 human-labeled dialogues, split 80/10/10 by conversation ID. Training uses LoRA r=16, alpha=32, dropout=0.05, learning rate 1e-4 with cosine decay, effective batch 64 on 8×A100 40GB GPUs, max sequence length 4,096, and gradient checkpointing. At step 800 the loss spikes from 0.62 to 2.10; batch logs show a cluster of 3,600-token examples against a 900-token median. By step 1,200 training loss is 0.38, validation loss is 0.91 and rising. Held-out exact match is 61%, but a fresh 2,000-example set from a new product line scores 34%. Invalid JSON appears in 18% of generations, mostly trailing commas and missing required fields. No labels exist for the new product line, and the product manager wants to ship in 10 days.
1. Diagnose the divergence between held-out and new-product-line performance, and explain the step-800 loss spike. What would you check before changing hyperparameters?
Read the worked answer
Start with data hygiene, because the 61% versus 34% gap has two plausible causes: overfitting and distribution shift. Splitting by conversation ID prevents same-conversation leakage but not template reuse or near-duplicates across IDs. Compute MinHash or embedding duplicates and cross-split n-gram overlap. If deduplication lowers held-out exact match, the original estimate was inflated. The new product line is covariate shift: new vocabulary, field names, and phrasing. A model that learned surface templates will fail there even without overfitting.
The rising validation loss alongside falling training loss is consistent with overfitting, but inspect validation labels first; label noise can also produce a 0.91 plateau. The step-800 spike coincides with 3,600-token examples, roughly four times the median. Check gradient norm, attention entropy, truncation, and whether the spike batch was packed incorrectly. If gradient norms spike, clip at 1.0 and use length-stratified batching before lowering the learning rate. Run ablations: early stopping near step 900, LoRA r=8 with dropout 0.1, and deduplicated training. For the new line, label 300 to 500 examples and build an error taxonomy. If OOD remains low after deduplication and early stopping, the fix is data coverage or retrieval, not more regularization.
2. The PM proposes adding grammar-constrained decoding to fix the 18% invalid JSON and shipping. Evaluate that proposal and define the evaluation metrics you would use.
Read the worked answer
Grammar-constrained decoding enforces syntax, not semantics. It can push schema validity close to 100% while leaving wrong field values, missing optional data, or unsafe payment instructions. It can also mask a capability gap and create an unnatural token distribution, so evaluate with and without constraints.
Define metrics at several levels: schema validity rate, exact match, field-level precision and recall, semantic equivalence, invalid-value rate, abstention rate, retry rate, and latency overhead. Test nested objects, optional fields, enums, and malformed user input. Add a post-validation retry with a bounded budget, but measure the added latency. Audit training targets for the same trailing-comma and missing-field errors; if labels are invalid, constrained decoding only hides the label problem.
The decision rule should be: schema validity above 99.5%, OOD human-rated correctness above 55% on at least 300 examples, and no regression on the existing product line. If constrained decoding fixes format but OOD exact match stays near 34%, shipping is still unsafe. Treat constrained decoding as a guardrail, not as evidence that the model understands the API schema.
3. Recommend a 10-day plan and go/no-go. Include what you would ship, to whom, and what evidence would change your mind.
Read the worked answer
Recommend no full launch. Days 1-2: deduplicate and audit labels; recompute held-out and OOD metrics. Days 3-4: early-stop checkpoint, length-stratified batching, gradient clipping, and a small LoRA-rank ablation. Days 5-6: add schema-constrained decoding and a validator with fallback. Days 7-8: label 500 new-line examples and test synthetic augmentation or retrieval of similar schemas. Days 9-10: canary behind a feature flag.
Ship only to internal users or opt-in customers on the existing product line, with schema validation, bounded retry, and fallback to a deterministic template or structured API. Gate: schema validity above 99.5%, old-line exact match at least 61%, new-line human correctness at least 55% on 300 examples, and zero unsafe payment actions in adversarial tests. Monitor invalid JSON, retry rate, escalation rate, and field-level errors.
Evidence that would change my mind: deduplication shows true held-out performance near 45%, new-line labels reveal a systematic missing field, or constrained decoding hides semantic errors. In those cases, delay and invest in data coverage. Document residual risk: new-line performance is estimated from a small sample and the fallback may not cover all intents.