CNLP key terms

Splitting raw text into smaller units called tokens, such as words, subwords, or characters, before any model can process it. The choice of tokenizer shapes vocabulary size, sequence length, and downstream accuracy.

A segmentation strategy that breaks rare words into frequent pieces while keeping common words whole, balancing vocabulary size against coverage of unseen words. It is the default for modern neural language models.

A data-driven subword algorithm that starts from characters and repeatedly merges the most frequent adjacent pair into a new symbol. The resulting merge table defines how any input string is segmented at inference time.

A subword algorithm similar to byte pair encoding but selecting merges by likelihood gain rather than raw frequency. It is the tokenizer used by BERT-family models and marks continuation pieces with a special prefix.

The policy a system uses when an input token is absent from its vocabulary, such as mapping it to an unknown token, decomposing it into subwords, or hashing it. Poor handling silently degrades accuracy on rare names and domain jargon.

The fixed set of tokens a model can represent, each mapped to an integer identifier. Its size trades off memory and embedding parameters against the average number of tokens needed per word.

Deleting very frequent function words such as articles and prepositions before classical feature extraction. It shrinks feature space but can destroy negation and phrase structure, so it is usually avoided for neural and sequence-labeling models.

Reducing a word to its dictionary base form using vocabulary and part-of-speech knowledge, so that 'running' becomes 'run'. It is linguistically motivated and more accurate but slower than simple suffix stripping.

A fast heuristic that chops word endings using rule patterns, often producing non-words such as 'studi'. It groups morphological variants for retrieval-style tasks but is cruder than dictionary-based normalization.

A weighting scheme that multiplies a term's frequency in a document by the inverse of the number of documents containing it, so common words count less. It converts text into sparse vectors usable by linear classifiers and search rankers.

A representation that counts token occurrences and ignores order, producing a sparse document vector. It is cheap and interpretable but loses word order, negation scope, and multiword meaning.

A contiguous sequence of n tokens, used to capture local word order in features or in a probabilistic language model. Larger n captures more context but multiplies sparsity and parameter count.

A model that assigns probabilities to sequences of tokens, usually by predicting the next token given preceding context. It underpins autocomplete, machine translation decoding, speech recognition rescoring, and generative assistants.

The exponentiated average negative log-likelihood a language model assigns to held-out text. Lower values mean the model is less surprised by real data, but scores are only comparable across models sharing the same tokenizer and vocabulary.

The cosine of the angle between two vectors, ranging from minus one to one and ignoring magnitude. It is the standard way to compare document, sentence, or embedding vectors in retrieval and clustering.

A probabilistic text classifier that applies Bayes' theorem while assuming features are conditionally independent given the class. Despite the false assumption it is a strong, fast baseline for spam filtering and topic tagging.

A linear model that predicts class probabilities through a sigmoid or softmax over weighted features. With sparse text features it remains a fast, interpretable, hard-to-beat baseline for many classification tasks.

A structured sequence-labeling model that scores whole label sequences and learns transition constraints between adjacent tags. It was the standard for named entity recognition before neural encoders, and it prevents invalid tag transitions.

Assigning each token a grammatical category such as noun, verb, or determiner. Tags feed downstream parsing, lemmatization, and entity extraction, and are usually produced by a sequence-labeling model over contextual representations.

Detecting spans of text that denote people, organizations, locations, dates, or domain-specific entities and labeling their type. It is a token-classification task central to resume parsing, compliance screening, and information extraction.

A dense low-dimensional vector learned for each token so that semantically similar words land near each other. Embeddings replace sparse counts as model input and enable similarity, clustering, and transfer learning.

A family of shallow neural models that learn word vectors by predicting a word from its neighbors, or neighbors from a word, over a large corpus. The resulting static vectors capture analogies but give one vector per word regardless of context.

A vector for a token computed from its surrounding sentence, so the same word receives different representations in different contexts. It resolves polysemy and is the output of transformer encoders used for transfer learning.

A learned weighting that lets each position gather information from other positions, with weights computed from query-key similarity. It provides content-based, parallelizable context mixing and is the core operation of transformer models.

Attention where queries, keys, and values all come from the same sequence, so every token aggregates information from the other tokens in that sequence. Stacked layers build deep contextual representations without recurrence.

A sequence model built from stacked blocks of self-attention and position-wise feed-forward layers with residual connections and normalization. Removing recurrence allows massive parallel training and long-range dependency modeling.

Information injected into token representations so a transformer can tell where each token sits in the sequence, since attention itself is order-blind. It may be fixed sinusoidal patterns or learned embeddings, and modern variants use rotary or relative schemes.

A pretrained encoder trained to predict randomly masked tokens using both left and right context. Fine-tuning its contextual representations set strong baselines for classification, question answering, and entity recognition.

A pretraining objective that hides a random subset of input tokens and trains the model to recover them from surrounding context. It forces bidirectional understanding and is the basis of encoder-only pretraining.

Producing text one token at a time, each conditioned on all previously generated tokens. Decoding choices such as greedy, beam, or sampling with temperature control creativity, repetition, and latency.

A transformer with billions of parameters pretrained on vast text corpora to predict the next token, then aligned with instruction tuning or preference optimization. It performs many tasks through prompting rather than task-specific training.

The maximum number of tokens a model can attend to in a single request, covering both input and generated output. Exceeding it forces truncation, chunking, or summarization, and longer windows raise memory and cost.

Designing instructions, examples, and output constraints in a model's input to elicit reliable behavior without changing weights. Techniques include few-shot exemplars, role framing, chain-of-thought cues, and structured output requests.

Placing a small number of worked input-output examples inside the prompt so the model infers the task pattern. It improves format adherence and accuracy for niche tasks but consumes context and can bias toward example artifacts.

Continuing training of a pretrained model on a smaller labeled dataset so its weights adapt to a specific task or style. It beats prompting when labeled data exists and behavior must be consistent and compact.

A pattern that retrieves relevant passages from an external index and inserts them into the prompt so the model answers from grounded evidence. It reduces hallucination and lets knowledge update without retraining weights.

Retrieval ranked by meaning similarity between embedded queries and documents rather than literal keyword overlap. It handles paraphrase and synonyms but needs a tuned embedding model and often hybrid scoring with lexical search.

A data structure mapping each term to the list of documents containing it, with positions and frequencies. It enables fast keyword lookup and scoring in search engines and underpins lexical retrieval baselines.

Fluent model output that is factually wrong or unsupported by any source. It arises because generation optimizes plausibility, not truth, and is mitigated with grounding, citations, abstention options, and human review.

Precision is the share of predicted positives that are correct; recall is the share of true positives that were found. Their harmonic mean, F1, summarizes both, and the trade-off should follow the cost of each error type.

F1 computed per class and then averaged with equal weight across classes, so rare classes matter as much as frequent ones. It is preferred over accuracy on imbalanced label sets where majority-class performance dominates.

A table of predicted versus actual class counts that reveals which classes are being mixed up. Inspecting it guides error analysis, threshold tuning, and targeted data collection better than a single aggregate score.

A translation metric that scores generated text against reference outputs by counting matching n-grams, with a penalty for overly short output. It correlates imperfectly with human judgment and ignores meaning and paraphrase.

Judging model outputs with annotators using defined rubrics such as relevance, fluency, and factuality. It is the ground truth for generative quality but needs clear guidelines, calibration, and inter-annotator agreement checks to be trustworthy.

A change in the statistical distribution of production inputs or outputs relative to the training data, degrading model accuracy over time. Monitoring feature and prediction distributions triggers retraining or rollback before quality visibly collapses.

Continuously tracking production metrics such as latency, error rates, prediction distributions, and business outcomes to detect degradation. It closes the loop between deployment and retraining and should include alerting thresholds and dashboards.

Running a new model alongside the live one on real traffic without serving its output to users, then comparing behavior offline. It exposes failures and latency issues before any user impact, at the cost of duplicated compute.

Splitting live traffic between a control and a variant model, then comparing a predefined business or quality metric with statistical testing. It establishes causal impact but requires enough traffic and careful guardrails against harm.

A versioned catalog of trained models with metadata on training data, hyperparameters, evaluation scores, and approval status. It makes deployments reproducible and auditable and supports rollback to a known-good artifact.

Adversarial probing of a system to surface harmful, biased, or unsafe outputs before release. For language models it includes jailbreak attempts, prompt injection, and edge-case prompts, with findings driving mitigation and guardrails.

The negative log-likelihood of the correct class under the model's predicted distribution, used as the standard training objective for classification and language modeling. Minimizing it pushes probability mass toward the observed targets.

Loss of previously learned capability when a model is trained on a new task or domain. It is mitigated by rehearsal data, parameter-efficient tuning, regularization, and evaluation suites that check old tasks after updates.