Natural Language Processing Fundamentals
Natural Language Processing Fundamentals tests whether you can describe how language is represented computationally and why ambiguity, rather than syntax alone, drives most design choices.
Levels of linguistic analysis
- Morphology: how words are built from roots and affixes; it determines whether stemming or lemmatization is appropriate.
- Lexical semantics: word sense. Bank (river vs finance) is polysemy, not a syntactic problem.
- Syntax: constituency and dependency structure; prepositional-phrase attachment ("I saw the man with the telescope") is genuine structural ambiguity.
- Pragmatics and discourse: reference, implicature, and coreference that span sentences.
Distributional and statistical foundations
The distributional hypothesis - words in similar contexts carry similar meaning - underlies embeddings. Zipf's law means a long tail of rare tokens dominates vocabulary coverage, which motivates subword tokenization and smoothing. Language modelling frames the field as P(token | context); classical pipelines chain tagging, parsing and classification, while neural systems train end-to-end.
Worked example
Task: decide whether "Bank of America raised rates" is a finance text. A keyword rule fires on bank, but bank is ambiguous, so the rule has low precision. A windowed context check (America, rates) resolves the sense; a contextual embedding would place bank near finance vectors here and near river in "the bank was muddy". The pipeline order is normalization, tokenization, sense-relevant features, then classification - and any stage can destroy evidence, as lowercasing does to US versus us.
Traps
- Treating tokenization as trivial; whitespace splitting fails on Chinese, hashtags, and "don't".
- Confusing stemming (crude stripping, studies to studi) with lemmatization (dictionary form, studies to study).
- Claiming a system "understands" when it models co-occurrence.
- Assuming more data fixes annotation noise or an ambiguous label definition.