Classical and Statistical NLP Methods
Classical and Statistical NLP Methods tests probabilistic sequence models, the assumptions behind them, and the failure modes those assumptions create.
Core models
- Naive Bayes: conditional independence assumption; use add-one (Laplace) smoothing or zero counts annihilate the product.
- HMM for tagging: states are tags, emissions are words, transitions include start and end markers; Viterbi decodes in O(T·S²).
- MEMM and CRF: discriminative sequence models. A CRF's global normalization avoids the label-bias problem, where an MEMM state with few outgoing transitions unfairly dominates the path.
- n-gram language models with Kneser–Ney smoothing; perplexity as an intrinsic measure.
Worked example
Spam versus ham with vocabulary V = 10. In spam, free appears 3 times among 20 tokens; in ham, once among 30. Add-one smoothing gives P(free|spam) = (3+1)/(20+10) = 0.133 and P(free|ham) = (1+1)/(30+10) = 0.05. With equal priors the likelihood ratio is about 2.67, favouring spam. Reasoning: the smoothing denominator adds V, the vocabulary size, not the number of classes, and unseen words contribute a small nonzero factor instead of a zero.
Traps
- Reporting per-token accuracy for named-entity recognition; entity-level F1 with exact span and type match is the metric that matters.
- Choosing an MEMM over a CRF without accounting for label bias.
- Comparing perplexities computed under different tokenizations or vocabularies; they are not comparable.
- Omitting start and end transitions in an HMM, which inflates sequence scores.