NVIDIA NCA-GENM cheat sheet

Scaled dot-product attention: Attention(Q,K,V) = softmax(QKᵀ/√d_k)·V; dividing by √d_k keeps logit variance near 1 as head dimension grows, preventing softmax saturation.

Decoder-only training objective: next-token cross-entropy L = −(1/N)Σ log p(x_t | x_<t); loss is reported per token, so values are comparable only across identical tokenizers.

Chinchilla compute-optimal allocation: about 20 training tokens per parameter, with loss following power laws in N (parameters), D (tokens), and compute C ≈ 6ND.

LoRA freezes W and learns ΔW = BA with rank r ≪ min(d,k); trainable parameters fall by roughly d·k / (r(d+k)), and adapters can be swapped at inference.

Mixture-of-Experts gating routes each token to its top-k experts, so total parameters grow while FLOPs per token stay near the dense k/n fraction.

DDPM forward process: q(x_t|x_0) = N(√ᾱ_t·x_0, (1−ᾱ_t)I); the network learns to predict noise ε, and sampling iterates the learned reverse process.

Classifier-free guidance: ε̂ = ε_∅ + s·(ε_c − ε_∅); s > 1 trades sample diversity for prompt adherence and can cause saturation or artifacts at high values.

Decision rule: use RAG when knowledge changes frequently or answers must cite sources; use fine-tuning when the gap is style, format, or task behavior.

Precision = TP/(TP+FP); recall = TP/(TP+FN); F1 = 2PR/(P+R). Report all three when class priors are skewed, since accuracy is dominated by the majority class.

ROC-AUC is threshold-free ranking quality; PR-AUC is the preferred summary when positives are rare (roughly under 10% prevalence).

Cosine similarity = (a·b)/(‖a‖‖b‖) ignores magnitude and suits normalized embeddings; Euclidean L2 is the default for unnormalized raw feature vectors.

Perplexity = exp(−(1/N)Σ log p(x_i)); lower is better and it is comparable only between models sharing the same tokenizer and vocabulary.

Leakage indicators: validation metrics far above production, duplicate or near-duplicate rows across splits, and features derived from the target; deduplicate before splitting.

KL divergence D_KL(P‖Q) = Σ P log(P/Q) is non-negative and asymmetric, so the direction (forward vs reverse) must always be stated.

Stratified splitting preserves class proportions; group splitting by source document or session prevents near-duplicate contamination in multimodal and text corpora.

RAG chunking trade-off: roughly 256–512 tokens per chunk improves retrieval precision; larger chunks retain context but dilute embedding specificity and waste prompt budget.

Three-way split: train for fitting, validation for early stopping and hyperparameter choice, test evaluated once for the final unbiased estimate.

Random search outperforms grid search when only a few hyperparameters matter; Bayesian/TPE search wins when each trial costs many GPU-hours.

Learning-rate schedule: linear warmup over the first 1–2% of steps then cosine decay is the transformer default; peak LR typically scales with global batch size.

Ablation = remove or replace one component and retrain with identical seeds, data order, and compute budget so metric deltas are attributable to that component.

Seed control: fix Python, NumPy, and framework seeds; deterministic kernels reduce throughput, so enable them only for comparison runs.

Early stopping patience counts validation evaluations without improvement; restore the best checkpoint by validation loss rather than the final step.

Report mean ± standard deviation over at least three seeds; a difference smaller than one standard deviation is not a demonstrated improvement.

BF16/FP16 mixed precision with loss scaling roughly halves activation memory and speeds training; confirm numerics against a short FP32 baseline.

CLIP-style InfoNCE loss maximizes cosine similarity for matched image-text pairs in a batch and minimizes it against the other N−1 in-batch pairs.

Cross-attention fusion: queries from one modality attend to keys/values from another; self-attention fuses tokens within a single modality.

ViT patch token count = (H/p)·(W/p); a 336×336 image with 14 px patches yields 576 tokens before projection into the language model.

Multimodal context budget = text tokens + visual tokens + audio tokens; prune, pool, or resize visual tokens before raising images-per-prompt.

Video sampling: a fixed frame count (typically 8–32) with uniform temporal stride keeps token cost constant regardless of clip duration.

Audio preprocessing: log-mel spectrograms, commonly 80 mel bins with 25 ms windows and 10 ms hop, feed conformer or Whisper-style encoders.

Modality gap: image and text embeddings occupy distinct regions of the joint space, so cross-modal similarity is not calibrated without explicit alignment training.

Data curation pipelines (NeMo Curator, Data Designer) deduplicate, filter by CLIP or aesthetic score, and synthesize captions before multimodal pretraining.

KV cache bytes ≈ 2 · layers · heads · head_dim · seq_len · batch · bytes_per_element (the factor 2 covers keys and values).

Arithmetic intensity = FLOPs / bytes moved; below the GPU ops:byte ratio a kernel is memory-bound, above it compute-bound, which decides whether fusion or tiling helps.

Tensor parallelism shards weight matrices across GPUs and needs an all-reduce per layer; pipeline parallelism splits layers and needs micro-batches to shrink the bubble.

FlashAttention tiles Q/K/V through on-chip SRAM and never materializes the N×N score matrix in HBM, reducing attention memory from O(N²) to O(N).

FP8 (E4M3/E5M2) with per-tensor scaling on Hopper or Blackwell roughly doubles throughput versus BF16 with small accuracy loss.

Speculative decoding: a draft model proposes k tokens and the target verifies them in one forward pass; speedup requires a high acceptance rate.

Throughput versus latency: continuous batching (TensorRT-LLM, Triton) raises tokens/s, but per-request latency rises as batch size grows.

Quantization decision: INT8/FP8 for memory- and latency-bound serving; keep BF16 for training and for tasks sensitive to long-tail or reasoning accuracy.

NVIDIA stack mapping: NeMo for training and customization, NeMo Retriever for RAG, NeMo Guardrails for safety, TensorRT-LLM and Triton for serving.

Container CUDA version must be ≤ the driver's supported CUDA version; nvidia-smi reports the driver and its maximum CUDA, nvcc --version the toolkit.

PyTorch: model.train()/model.eval() toggle dropout and batch-norm behavior; wrap generation and evaluation in torch.inference_mode() for lower memory.

A run is reproducible only if config, code commit hash, dataset version, and random seeds are all recorded with the checkpoint.

Prompt templates are code: version them, keep system/user/assistant roles explicit, and validate structured outputs against a schema.

LLM API clients need retries with exponential backoff and jitter on 429/5xx, explicit timeouts, and idempotency keys for side-effecting calls.

CI gates: unit tests for preprocessing, a 10-sample smoke test of the training loop, and a fixed-prompt regression test on generation outputs.

Store weights in safetensors rather than pickle (no arbitrary code execution) and keep tokenizer files and config alongside the checkpoint.

Hallucination controls: retrieval grounding with citations, abstention/refusal paths, and constrained decoding or schema validation on structured outputs.

NeMo Guardrails applies input, dialogue, and output rails (jailbreak, topic control, PII, fact-checking) as Colang flows around the base model.

Bias measurement compares outcome rates across subgroups; a disparate impact ratio below 0.8 (four-fifths rule) is commonly treated as evidence of adverse impact.

PII handling: detect and redact before logging or training; k-anonymity and differential privacy with a stated ε bound limit re-identification risk.

NIST AI RMF organizes work into Govern, Map, Measure, and Manage functions; it is a voluntary risk framework, not a certification.

EU AI Act risk tiers: unacceptable (prohibited practices), high-risk (conformity assessment and documentation), limited-risk (transparency duties), minimal-risk.

Model cards record intended use, training data, per-subgroup evaluation metrics, and known limitations; they support audit and do not by themselves establish legal compliance.

Red teaming probes prompt injection, jailbreaks, and data exfiltration before release; every confirmed finding becomes a regression test in the evaluation suite.