CNLP flashcards

32 cards. Read the term and the definition here. The deck at the top of this page is optional.

Zipf's law

Token frequency is inversely proportional to rank, so a few words cover most tokens and many rare words appear once. This motivates subword tokenization and frequency cutoffs.

Perplexity

Exponentiated average negative log-likelihood per token. Lower values indicate better fit, but it is comparable only across models with the same tokenization and vocabulary.

Lemmatization vs stemming

Lemmatization uses a lexicon and part-of-speech tag to return a valid base form. Stemming heuristically strips suffixes and may produce non-words such as 'studi'.

Subword tokenization

Algorithms such as BPE, WordPiece, and Unigram split text into frequent units, balancing vocabulary size and sequence length while limiting out-of-vocabulary tokens.

TF-IDF

Term frequency multiplied by inverse document frequency, idf = log(N/df). It downweights terms common across documents and rows are usually L2-normalized for cosine similarity.

BM25

A ranking function combining term frequency saturation (k1) with document-length normalization (b). It typically outperforms plain TF-IDF for lexical retrieval.

Feature hashing

Maps features into a fixed number of buckets using a hash function, giving constant memory and streaming support at the cost of collisions and interpretability.

Stopword removal tradeoff

Removing frequent words shrinks feature space but deletes negation and cue phrases. It harms sentiment and inference tasks, so it is usually skipped for transformer inputs.

Naive Bayes classifier

A generative classifier assuming conditional independence of features given the class. Multinomial NB with Laplace smoothing is a fast, strong baseline for sparse text.

Conditional Random Field

A discriminative sequence model scoring the whole label sequence with transition and emission features. It avoids the HMM label-bias problem and is trained by maximizing conditional likelihood.

Viterbi algorithm

Dynamic programming that finds the highest-probability label sequence for an HMM in O(T times the square of the number of states) for T tokens.

Entity-level F1

Sequence-labeling metric that counts a prediction correct only when the full entity span and type match the gold annotation, unlike token-level accuracy.

LDA topic model

A generative model where documents mix topics and topics mix words, governed by Dirichlet priors. Inference uses collapsed Gibbs sampling or variational expectation maximization.

Scaled dot-product attention

Attention weights equal softmax of Q times K transpose divided by the square root of the key dimension, then multiplied by V. Scaling stabilizes gradients for large dimensions.

Layer normalization placement

Pre-LN applies normalization before each sublayer and trains more stably without warmup; post-LN applies it after the residual, matching the original transformer.

LoRA

Low-rank adaptation freezes base weights and trains small rank-decomposed matrices added to them, updating under one percent of parameters with little quality loss.

Masked language modeling

Pretraining objective that hides a fraction of tokens and predicts them from bidirectional context, producing encoder representations used for classification and extraction.

Rotary position embeddings

RoPE rotates query and key vectors by position-dependent angles so dot products encode relative offsets, enabling length extrapolation in many decoder-only models.

Top-p sampling

Nucleus sampling keeps the smallest token set whose cumulative probability reaches p, then samples within it. It adapts candidate count to distribution shape.

Temperature scaling

Divides logits by T before softmax. Values below one sharpen the distribution toward likely tokens; values above one flatten it and increase diversity.

Chinchilla scaling

Compute-optimal training allocates roughly twenty tokens per model parameter, so many models are undertrained relative to their size under fixed compute budgets.

Retrieval-augmented generation

Retrieves relevant passages from an external index and conditions generation on them, improving factuality and allowing citations without retraining the model.

KV cache

Stores past key and value tensors during autoregressive decoding to avoid recomputation. Memory grows linearly with context length and batch size.

Micro vs macro averaging

Micro-average pools counts across classes and weights by support; macro-average averages per-class metrics equally, exposing poor performance on rare classes.

PR-AUC

Area under the precision-recall curve, equivalent to average precision. It is more informative than ROC-AUC when the positive class is rare.

LLM-as-judge

Using a language model to score outputs against a rubric. It scales evaluation but requires calibration against human labels and checks for position and verbosity bias.

Model card

A document recording intended use, training data, evaluation results, limitations, and ethical considerations for a released model, supporting transparency and governance.

Shadow deployment

A new model runs on live traffic without serving responses while its outputs are compared to the incumbent, catching regressions before any user impact.

Data drift detection

Monitoring shifts in input distributions using statistics such as population stability index or KL divergence on token or embedding distributions, triggering retraining alerts.

Golden-set regression test

A fixed labeled evaluation set run in CI on every model or preprocessing change, with pass thresholds that block deployment when quality degrades.

RAG chunking strategy

Splitting documents into passages of roughly 200 to 500 tokens with overlap, preserving section boundaries so retrieved context stays coherent and citable.

Human-in-the-loop routing

Sending low-confidence predictions to human reviewers while auto-approving high-confidence ones, balancing automation rate against error cost.