CNLP cheat sheet

Tokenization granularity determines vocabulary size and sequence length: word-level gives large sparse vocabularies, subword (BPE/WordPiece/Unigram) balances coverage and length, character-level minimizes OOV but lengthens sequences.

Zipf's law: token frequency is inversely proportional to rank, so a small number of words covers most tokens; this motivates frequency cutoffs and subword models for rare words.

Morphology: agglutinative languages (Turkish, Finnish) pack many morphemes per word, inflating OOV rates for word-level models and favoring morpheme-aware or subword tokenization.

Bag-of-words assumes exchangeability of tokens, discarding order; n-gram models reintroduce local order with a Markov assumption of order n-1.

Perplexity = exp(-(1/N) * sum log P(w_i | context)); lower is better and it is comparable only across models with identical tokenization and vocabulary.

Ambiguity types: lexical (bank), syntactic (PP attachment), semantic (scope), and pragmatic (intent); resolution requires context, which motivates contextual embeddings.

Regular vs irregular inflection, stemming vs lemmatization: lemmatization uses a lexicon and POS tag to return a valid base form; stemming applies heuristic suffix stripping and may yield non-words.

Evaluation-oriented task taxonomy: sequence labeling (NER, POS), sequence classification (sentiment), seq2seq (translation, summarization), and structured ranking (QA, retrieval).

Normalization order matters: Unicode NFKC normalization, then lowercasing (case-folding), then punctuation/diacritic handling; aggressive lowercasing harms cased tasks like NER.

Stopword removal reduces dimensionality but destroys negation and cue phrases ('not', 'no', 'but'); skip it for sentiment, NLI, and transformer inputs.

TF-IDF: tf(t,d) * idf(t) with idf = log(N / df(t)) or smoothed log((1+N)/(1+df))+1; L2-normalize rows so cosine similarity equals dot product.

Sublinear TF: 1 + log(tf) dampens the effect of repeated terms; BM25 adds document-length normalization and a saturation parameter k1 (typ. 1.2-2.0) and b (typ. 0.75).

Hashing trick (feature hashing) maps features to a fixed 2^b bucket space in O(1) memory but loses interpretability and introduces collisions.

Character and word n-grams capture morphology and local phrasing; char n-grams (3-5) are robust to typos and OOV in log-linear text classifiers.

Class imbalance handling: class weights, resampling, or threshold tuning on the validation set; report precision, recall, and F1 per class rather than accuracy alone.

Embedding initialization for text classifiers: pretrained static vectors (word2vec/GloVe/fastText) cover rare words via subwords (fastText) but are context-independent and polysemy-blind.

Naive Bayes: P(c|d) proportional to P(c) * product P(w|c); multinomial NB with Laplace smoothing alpha=1 is a strong sparse-text baseline.

Logistic regression on TF-IDF is a robust baseline for text classification; L2 regularization controls overfitting, and coefficients are interpretable per feature.

Linear SVM maximizes margin; for linearly separable high-dimensional sparse text it often matches or beats logistic regression; hinge loss is sensitive to class imbalance.

Hidden Markov Models for POS tagging: transition P(tag_i|tag_{i-1}) and emission P(word|tag); Viterbi decoding is O(T * |S|^2) for T tokens and |S| states.

CRFs model P(y|x) globally with feature functions over label transitions, avoiding the HMM label-bias problem; training maximizes conditional log-likelihood (requires forward-backward).

Conditional independence in NB makes probability estimates poorly calibrated even when ranking (and hence accuracy) is decent; calibrate with Platt scaling if probabilities are needed.

Evaluation for sequence labeling uses entity-level (exact span) precision/recall/F1 via conlleval-style scoring, not token-level accuracy, which overstates performance on O tags.

Latent Dirichlet Allocation: document-topic Dirichlet prior alpha, topic-word prior beta; inference via collapsed Gibbs sampling or variational EM; choose K by held-out perplexity or coherence.

RNN/LSTM gradient flow: LSTM gates (input, forget, output) mitigate vanishing gradients; GRU merges gates and has fewer parameters, often comparable quality.

Attention: Attention(Q,K,V) = softmax(QK^T / sqrt(d_k)) V; the sqrt(d_k) scaling keeps logits in a stable range for large head dimensions.

Transformer encoder layer: multi-head self-attention, residual + LayerNorm, position-wise FFN, residual + LayerNorm; pre-LN (norm before sublayer) improves training stability.

Positional information: sinusoidal encodings are fixed; learned absolute embeddings are parameterized; RoPE rotates query/key pairs so relative position is encoded by dot products.

Pretraining objectives: masked language modeling (BERT-style, bidirectional encoder) vs causal language modeling (GPT-style, autoregressive decoder) vs span denoising (T5).

Fine-tuning recipe: low learning rate (1e-5 to 5e-5), linear warmup (5-10% of steps), AdamW with weight decay 0.01, and early stopping on validation metric.

Parameter-efficient tuning: LoRA adds low-rank matrices B*A to frozen weights (rank r typically 4-64), training <1% of parameters; adapters insert bottleneck layers.

Regularization for small NLP datasets: dropout (0.1-0.3), label smoothing (0.1), weight decay, and data augmentation (back-translation, token dropout).

Decoder-only LLM generation is autoregressive: sample or argmax next token, append to context, repeat; max_new_tokens bounds cost, not the model's context window.

Decoding controls: temperature T scales logits (T<1 sharpens, T>1 flattens); top-k truncates to k tokens; top-p (nucleus) keeps the smallest set with cumulative probability >= p (typ. 0.9-0.95).

Scaling laws: loss falls as a power law in parameters, data, and compute (Chinchilla: ~20 tokens per parameter for compute-optimal training).

Context window limits attention cost: full self-attention is O(n^2) in sequence length; FlashAttention, sliding-window, and sparse attention reduce memory/time.

Instruction tuning and RLHF/DPO align base models to follow instructions and preferences; RLHF uses a reward model plus PPO, DPO optimizes directly on preference pairs.

Prompt engineering patterns: few-shot exemplars, chain-of-thought prompting, role/system prompts, and structured output schemas; retrieval augmentation grounds answers in external documents.

Hallucination mitigation: retrieval-augmented generation, citation requirements, temperature reduction, and self-consistency or verifier checks; none eliminates hallucination entirely.

Quantization (INT8/INT4) and distillation reduce inference cost; KV-cache reuse speeds generation but grows linearly with context length and batch size.

Classification metrics: precision = TP/(TP+FP), recall = TP/(TP+FN), F1 = 2PR/(P+R); macro-average weights classes equally, micro-average weights by support.

For imbalanced text classification, PR-AUC (average precision) is more informative than ROC-AUC because it focuses on the minority positive class.

LLM evaluation dimensions: accuracy, faithfulness, relevance, toxicity, and latency/cost; use human review, LLM-as-judge with calibration, and task-specific benchmarks.

Bias sources: training data skew, labeler bias, and proxy features; measure with disaggregated metrics across demographic or domain subgroups.

Privacy: PII detection/redaction before training and logging; differential privacy adds calibrated noise; membership inference and extraction attacks target memorized training data.

Explainability: attention weights are not reliable explanations; use SHAP/LIME for local attributions on classical models and counterfactual or ablation tests for neural models.

Data governance: consent, purpose limitation, retention limits, and documented data lineage; model cards and datasheets record intended use and limitations.

Regulatory framing: risk-tiered obligations (e.g., transparency, human oversight, impact assessments) apply to high-risk NLP uses such as hiring or credit decisions.

Serving patterns: batch scoring for offline analytics, real-time REST/gRPC for interactive use, and streaming (Kafka) for high-throughput pipelines; latency budgets drive the choice.

Model compression: quantization, pruning, distillation, and ONNX/TensorRT export reduce latency; measure p50/p95/p99 latency, not just averages.

Versioning: pin model artifact, tokenizer, preprocessing code, and config together; a tokenizer mismatch silently degrades quality.

Monitoring: track input drift (PSI, KL divergence on token distributions), output drift, error rates, latency, and business KPIs; alert on thresholds.

CI/CD for ML: unit tests on preprocessing, golden-set regression tests for model quality, canary or shadow deployment, and automated rollback.

Caching and batching: KV-cache reuse, request batching, and embedding caches reduce cost; set timeouts and circuit breakers to protect upstream services.

Cost controls: token counting per request, context truncation, model routing (small model first, escalate on low confidence), and rate limiting.

Fallback and safety: guardrails for PII, prompt injection, and unsafe outputs; deterministic fallback responses when the model or dependency fails.

Use-case selection: map the business problem to a task type (classification, extraction, retrieval, generation) and define a measurable success metric before modeling.

Human-in-the-loop design: route low-confidence predictions to reviewers; track reviewer agreement (e.g., Cohen's kappa) to validate label quality.

RAG for enterprise Q&A: chunk documents (typ. 200-500 tokens with overlap), embed, index in a vector store, retrieve top-k, then generate with citations.

Information extraction: span labeling for entities, relation extraction for links, and schema validation before writing to downstream systems.

Conversational agents: intent classification plus slot filling for task-oriented bots; guardrails, fallback intents, and escalation to humans for low confidence.

ROI framing: quantify cost per request, deflection rate, time saved per ticket, and error cost; compare against a manual baseline.

Data flywheel: log predictions and outcomes, label a sample, retrain on a schedule, and A/B test against the incumbent model before promotion.

Localization and compliance: language-specific tokenization, locale-aware normalization, and content policy checks per market before launch.