Data Analysis
What to know
LLM data work is a pipeline: collect, extract text, filter, deduplicate, decontaminate, and mix. Quality filters are usually cheap heuristics first - document length, symbol-to-word ratios, stop-word presence, repeated n-gram fraction, language ID confidence - then model-based filters such as perplexity scoring or classifier scores from a small labeled set.
Deduplication matters more than most candidates expect. Exact duplicates fall to hashing; near-duplicates need MinHash + LSH or embedding similarity. Decontamination is different: you scan training data for overlap with evaluation benchmarks (commonly 8- to 13-gram matches) and remove it, otherwise benchmark scores are inflated and meaningless.
PII and toxicity scrubbing are separate passes with their own recall targets. Synthetic data (self-instruct, distillation from a stronger model) is now standard, but it must be filtered and deduplicated too, or model collapse and repetition creep in.
Data mixture weights across domains are hyperparameters. Track token counts, not document counts, and know that English runs roughly 1.3 tokens per word for common BPE vocabularies.
When humans label data, report agreement: Cohen's kappa corrects for chance agreement, raw percent agreement does not.
Traps
- Deduplicating only within a split, missing cross-split leakage.
- Confusing deduplication (redundancy) with decontamination (benchmark leakage).
- Reporting word counts as token counts, or ignoring padding tokens in budget math.
- Sampling a skewed source and calling it representative.
Worked example
A 1B-token budget at ~1.3 tokens/word is about 770M English words. If your filtered corpus yields 400 words/document on average, you need roughly 1.9M documents, before any upsampling of scarce high-quality domains.