CNLP: Text Preprocessing and Feature Engineering

Text Preprocessing and Feature Engineering tests whether you can turn raw strings into reproducible, leakage-free model inputs that behave identically in training and serving.

Normalization and tokenization

• Unicode NFKC normalization, case folding, and accent handling must be applied identically at train and inference time.
• Tokenizer families: regex/whitespace, BPE (merge frequent character pairs), WordPiece (likelihood-based merges), unigram (probabilistic vocabulary pruning).
• Prefer lemmatization over aggressive stemming when meaning matters; never strip negation words ("not", "no") before sentiment features.

Features

• Bag-of-words, n-grams, and TF-IDF: tf-idf(t,d) = tf(t,d) × idf(t) with idf = ln(N/df) or the smoothed ln((1+N)/(1+df)) + 1.
• The hashing trick bounds memory without a stored vocabulary; subword vocabularies reduce out-of-vocabulary loss.
• L2-normalizing document vectors keeps long documents from dominating similarity.

Worked example

Three documents: d1 "cat sat mat", d2 "dog sat", d3 "cat ran". For cat: df = 2, N = 3, so idf = ln(3/2) ≈ 0.405. Raw term frequency in d1 is 1, giving tf-idf ≈ 0.405. Reasoning: inverse document frequency automatically down-weights terms spread across many documents - which is why the scores near zero even without a stopword list. Length normalization would then divide by the vector norm.

Traps

• Fitting the vectorizer or IDF on the full dataset before splitting; this leaks test information and inflates scores.
• Inconsistent preprocessing between training and serving, producing silent production drift.
• Truncating at 512 tokens without checking where the label-bearing evidence actually sits.
• Assuming subword token count equals word count; it is often 1.3 to 1.5 times larger.