NVIDIA GenAI LLM flashcards

32 cards. Read the term and the definition here. The deck at the top of this page is optional.

Scaled dot-product attention

Attention computes softmax(QK^T divided by the square root of the key dimension) times V. The scaling keeps dot products from growing large and saturating softmax.

KV cache

A KV cache stores previously computed keys and values during autoregressive generation so each new token avoids recomputing past attention states, trading memory for speed.

LoRA

Low-Rank Adaptation freezes the pretrained weight matrix and learns a low-rank update, usually ΔW=BA, reducing trainable parameters and optimizer memory during fine-tuning.

Quantization

Quantization maps model weights or activations from higher precision such as FP16 to lower precision such as INT8 or INT4, reducing memory and bandwidth but sometimes lowering accuracy.

Temperature

Temperature divides logits before softmax. Values below one sharpen the token distribution and make output more deterministic; values above one flatten it and increase diversity.

Retrieval-augmented generation

RAG retrieves relevant external documents and inserts them into the model context before generation, improving factual grounding without changing the model weights.

Perplexity

Perplexity is the exponentiation of average cross-entropy loss per token. Lower values indicate better prediction, but comparisons require the same tokenizer and dataset.

Data leakage

Data leakage occurs when information from the target or test set influences training or preprocessing, producing overly optimistic evaluation results that do not generalize.

Stratified split

A stratified split preserves the proportion of each class or group in every data partition, which is important for imbalanced classification and small datasets.

Precision

Precision is true positives divided by true positives plus false positives. It measures how many predicted positive cases are actually positive.

Recall

Recall is true positives divided by true positives plus false negatives. It measures how many actual positive cases the model successfully finds.

PR-AUC

Precision-recall area under the curve summarizes ranking performance across thresholds and is often more informative than ROC-AUC when positive cases are rare.

Data drift

Data drift is a change in input feature distributions between training and production. It is detected by comparing distributions with statistics such as PSI or KS tests.

p-value

A p-value is the probability of observing data at least as extreme as the collected data, assuming the null hypothesis is true. It is not the probability that the null is true.

Statistical power

Statistical power is the probability of detecting a true effect of a given size when it exists. It increases with sample size and effect size and decreases with variance.

Confidence interval

A confidence interval gives a range of plausible values for an estimated effect, combining point estimate and uncertainty to support practical significance decisions.

Multiple comparisons

Testing many hypotheses at once inflates the chance of false positives. Corrections include Bonferroni adjustment of alpha or false discovery rate control.

LLM-as-judge

An LLM-as-judge uses a language model to score or compare outputs against a rubric. It requires calibration and controls for position and verbosity bias.

Sequential testing

Sequential testing analyzes results repeatedly as data arrive. Valid methods adjust thresholds to control Type I error, unlike repeated peeking at a fixed-horizon test.

CUDA

CUDA is NVIDIA's parallel computing platform and programming model for running kernels on GPUs, widely used to accelerate deep learning training and inference.

TensorRT

TensorRT is an NVIDIA inference optimizer that applies layer fusion, precision calibration, and kernel autotuning to produce fast GPU engines for trained models.

Triton Inference Server

Triton Inference Server serves models through HTTP and gRPC endpoints, supporting dynamic batching, multiple frameworks, and versioned model repositories.

NCCL

NCCL is NVIDIA's library for multi-GPU and multi-node communication collectives such as all-reduce and broadcast, enabling efficient distributed training.

Mixed precision

Mixed precision uses FP16 or BF16 for most computation while keeping FP32 master weights and using loss scaling to maintain numerical stability during training.

Gradient accumulation

Gradient accumulation sums gradients over several micro-batches before an optimizer step, simulating a larger batch when GPU memory is limited.

NVIDIA NIM

NVIDIA NIM provides packaged, optimized inference microservices for deploying generative AI models with standard APIs and GPU-accelerated runtimes.

Demographic parity

Demographic parity requires positive prediction rates to be equal across protected groups. It ignores actual outcome rates and can conflict with accuracy.

Equalized odds

Equalized odds requires equal true positive and false positive rates across protected groups, conditioning on the actual outcome rather than only prediction rates.

Differential privacy

Differential privacy adds calibrated noise to computations so that any single record's influence is bounded. Smaller epsilon gives stronger privacy but lower utility.

Prompt injection

Prompt injection is an attack where untrusted input contains instructions that override or manipulate the model's intended behavior, often to exfiltrate data or bypass guardrails.

Model card

A model card documents a model's intended use, training data, evaluation metrics, limitations, and ethical considerations to support transparent and responsible deployment.

Red teaming

Red teaming is structured adversarial testing that probes a model or system for harmful, unsafe, or unintended behavior before and after deployment.