The 18% Perplexity Win That Broke Citation Accuracy
A team fine-tunes a 7B decoder to summarize legal filings. To shorten context they replace the 32k BPE tokenizer with a 50k vocabulary trained on 40M pages of their own corpus; the average document drops from 3,100 to 2,050 tokens. Validation loss falls from 1.62 to 1.44 nats/token and the result is presented to leadership as "perplexity down 18%". Training used the same 9B-token budget, the same 16 A100s, and the same schedule. Three weeks after launch the extractive QA eval on a 400-item held-out set drops from 71.2% to 66.8%, and answers citing a page number that does not exist rise from 1.1% to 3.4%. The product SLA requires bad-citation rate under 1.5%. A full retrain costs one week on 16 GPUs; rolling back to the previous checkpoint takes two hours. The 40M-page tokenizer corpus was collected by the same crawler that produced the eval set.
1. Leadership asks you to confirm the "18% perplexity improvement". Normalize the number and state what it can and cannot support.
Read the worked answer
Per-token loss is not comparable across tokenizers: a token now covers a different amount of text, and the softmax is over a different vocabulary size. The comparable quantity is bits per byte, loss_nats / ln(2) * tokens / bytes.
Using the document-level ratio (bytes cancel out):
- old: 1.62 nats x 3,100 tokens = 5,022 nats per document
- new: 1.44 nats x 2,050 tokens = 2,952 nats per document
- ratio = 0.59, i.e. roughly 41% fewer bits per document
So the honest statement is bigger than the one leadership was given, and that is the problem. A tokenizer swap plus a fresh embedding table cannot plausibly buy 41% compression on held-out text. An effect that large is a contamination signal: the new BPE was trained on 40M pages from the same crawler that built the eval set, so the validation text is now closer to in-distribution than it was.
The number supports one claim only: the model compresses this corpus better. It says nothing about citation grounding, which is what the SLA measures. Do not defend or repeat the 18%.
2. You have one week of 16-GPU time but do not want to spend it yet. What do you do in the first day, and what single controlled experiment would you authorize afterwards?
Read the worked answer
Day one is cheap diagnostics, not training.
- Suspect the harness first. A vocabulary change alters whitespace and digit handling; prompts and stop sequences tuned for the old tokenizer can silently degrade extraction. Re-run the 400-item eval with a re-tuned prompt and decode settings (a few GPU-hours) before blaming the model.
- Quantify overlap. Deduplicate the eval set against the 40M-page tokenizer corpus. If n-gram overlap is non-trivial, the loss numbers are void and the QA drop is the only trustworthy signal.
- Inspect the failures. 3.4% vs 1.1% bad citations is 9-10 items out of 400. Read them. If they cluster on page-number and date spans, the cause is tokenization of digit runs, not general quality.
The experiment to authorize is a proxy pair, not a full retrain: two ~1B models at 2-3% of the token budget, identical data and schedule, one with each tokenizer, scored on de-duplicated held-out text by bits-per-byte plus the QA eval. That isolates the tokenizer as a variable for a fraction of a week of GPUs, and it tells you whether to invest the full week at all.
3. What is your decision today, and what would have to be true before the new tokenizer can ship?
Read the worked answer
Roll back today. The SLA is a hard contract at 1.5% bad citations; production is at 3.4%, roughly 2.3x over, and rollback costs two hours. Nothing learned this week changes that arithmetic, and every hour of exposure is uninsured risk. Rolling back does not throw away the tokenizer work - the old checkpoint has no dependency on it.
To ship later, all of the following must hold:
- Bits-per-byte on a de-duplicated held-out set beats the incumbent tokenizer, computed on identical text.
- The eval set is verified disjoint from the tokenizer training corpus.
- The QA and bad-citation metrics are re-baselined for the new tokenizer - the prompt, stop conditions and any cached prefixes all need retuning, so a raw comparison against the old harness is invalid in both directions.
- A full retrain reaches the same token budget, because the shorter sequences mean fewer optimizer steps see the same documents; a token-budget-equal run is not the same as a document-equal run.
Also flag the one-way door: a tokenizer change invalidates cached KV prefixes, existing fine-tunes, and any downstream embeddings. It should be treated as a migration with a freeze window, not a checkpoint swap.