Machine Learning and Deep Learning for NLP
Machine Learning and Deep Learning for NLP tests how learned representations replaced sparse features, and which training decisions make or break a model.
Representations
- word2vec: skip-gram predicts context from a target word, CBOW predicts the target from context; negative sampling replaces the full softmax.
- GloVe factorizes global co-occurrence counts; FastText composes subword n-grams, giving robustness to out-of-vocabulary forms and morphology.
- RNNs, LSTMs and GRUs encode sequences but suffer vanishing gradients and sequential latency; CNNs capture local n-gram patterns; attention removes recurrence entirely.
Training mechanics
- Teacher forcing during training versus free-running decoding at inference creates exposure bias.
- Gradient clipping, learning-rate warmup, dropout, and class weighting are standard stabilizers.
- Fine-tuning adapts pretrained weights; feature extraction freezes them and trains only a classifier head.
Worked example
A 50,000-token vocabulary with 300-dimensional embeddings holds 15M embedding parameters; an untied output softmax adds another 15M. Reasoning: the output layer dominates cost, so weight tying, sampled softmax, or negative sampling with roughly 5 to 20 negatives is used - these approximate the full softmax gradient at a fraction of the compute.
Traps
- Lowercasing text before using cased embeddings, destroying US versus us distinctions.
- Using pretrained embeddings with no OOV policy, silently mapping rare words to
<unk>. - Comparing against only a majority-class baseline while ignoring per-class recall.
- Repeatedly tuning on the validation set, then reporting that score as a test result.