NVIDIA GenAI LLM cheat sheet

Scaled dot-product attention is Attention(Q,K,V)=softmax(QK^T/√d_k)V; dividing by √d_k keeps logits small and avoids softmax saturation.

A causal mask sets future-position logits to -infinity before softmax, so token t can attend only to positions ≤ t during autoregressive training.

KV caching stores past keys and values to avoid recomputation; memory grows with layers × heads × sequence length × head dimension × precision.

LoRA freezes base weights and learns a low-rank update ΔW=BA; trainable parameters scale with r(d_in+d_out) rather than d_in×d_out.

Quantization maps FP16/FP32 weights or activations to INT8/INT4; it reduces memory and bandwidth but requires calibration and can reduce accuracy.

Temperature T divides logits before softmax: T<1 sharpens the distribution, T>1 flattens it; top-p and top-k truncate the candidate set.

Retrieval-augmented generation conditions generation on retrieved chunks; quality depends on chunking, embedding model, retrieval recall, and reranking.

Perplexity equals exp(cross-entropy loss); lower is better only when comparing models on the same tokenizer and evaluation corpus.

Split data into train, validation, and test sets; fit preprocessing statistics such as scaling and imputation on the training set only to prevent leakage.

Stratified splitting preserves class proportions across folds and is important for imbalanced classification and small datasets.

With class imbalance, accuracy is misleading; report precision, recall, F1, PR-AUC, and the confusion matrix.

Precision = TP/(TP+FP), recall = TP/(TP+FN), and F1 is the harmonic mean of precision and recall; the decision threshold trades one for the other.

ROC-AUC measures ranking quality across thresholds; PR-AUC is more informative when positive cases are rare.

k-fold cross-validation averages performance over k folds; use grouped or time-series splits when observations are correlated over time or entities.

Data drift compares training and production feature distributions using statistics such as PSI, KL divergence, or KS tests.

Deduplicate and near-deduplicate before splitting; duplicates across train and test inflate measured performance.

For English text, token count is often approximated as characters/4; estimate context-window fit and cost from token counts, not word counts.

Define the hypothesis, primary metric, alpha, and power before running a test; the p-value is P(data|H0), not P(H0|data).

Required sample size increases with variance and decreases with effect size; power of 0.8 and alpha of 0.05 are common defaults.

A confidence interval gives an effect range; practical significance requires comparing the effect size with a business or safety threshold.

Multiple comparisons inflate false positives; control family-wise error with Bonferroni alpha/m or use false discovery rate methods.

Repeated peeking during a fixed-horizon test inflates Type I error; use a fixed analysis time or a valid sequential test.

A valid A/B test requires random assignment, stable unit treatment, guardrail metrics, and no cross-group interference.

LLM offline evaluation uses golden sets with exact match, F1, BLEU/ROUGE, embedding similarity, and rubric-based LLM judges.

LLM judges can show position, verbosity, and self-preference bias; randomize order and calibrate against human-labeled examples.

Reproducibility requires fixed seeds, temperature, model version, prompt template, decoding parameters, and logged evaluation data.

CUDA is NVIDIA's parallel-computing platform for GPU kernels; cuDNN provides optimized deep-learning primitives such as convolution and attention.

TensorRT optimizes trained models for inference through layer fusion, precision calibration, kernel autotuning, and engine serialization.

Triton Inference Server serves models through HTTP/gRPC endpoints with dynamic batching and a versioned model repository.

NCCL provides multi-GPU and multi-node collectives such as all-reduce and broadcast for distributed training.

Mixed precision uses FP16 or BF16 compute with FP32 master weights and loss scaling to preserve numerical stability.

Gradient accumulation simulates a larger batch; effective batch size = micro-batch × accumulation steps × data-parallel replicas.

NVIDIA NeMo supports conversational AI and LLM training; NVIDIA NIM packages optimized inference microservices.

RAPIDS accelerates dataframes and classical ML on GPU with cuDF and cuML; DALI builds high-throughput input pipelines.

Containerize with pinned CUDA and driver versions, test on the target GPU, and log GPU model, driver, and library versions.

Fairness criteria include demographic parity, equalized odds, and equal opportunity; they generally cannot all hold simultaneously except in trivial cases.

PII must be detected and redacted before training or logging; apply data minimization, access control, and retention limits.

Differential privacy adds calibrated noise; epsilon bounds privacy loss, and smaller epsilon means stronger privacy but lower utility.

SHAP provides feature-attribution explanations; attention maps are not by themselves faithful explanations of model reasoning.

Robustness testing covers adversarial inputs, distribution shift, prompt injection, jailbreaks, and out-of-distribution queries.

Hallucination mitigation includes retrieval grounding, citations, abstention, output validation, and human review for high-risk use.

A model card documents intended use, training data, evaluation metrics, limitations, and ethical considerations.

Red teaming probes harmful or unsafe behavior; guardrails filter inputs and outputs, and audit logs support governance and incident review.