NVIDIA GenAI LLM common mistakes
NVIDIA GenAI LLM · Understand the traps and how to avoid them.
When asked how to make the model answer questions about a frequently updated internal document set, the candidate selects 'fine-tune the LLM on the documents' instead of retrieval-augmented generation.
They collapse all 'make the model know X' problems into fine-tuning, because fine-tuning is the most familiar adaptation lever and the exam's scenario text mentions 'our own data'. They do not separate parametric knowledge (weights) from non-parametric knowledge (retrieved context).
What to do instead
Apply a three-way decision rule before answering: changing facts / per-user or per-tenant data / needs citations → retrieval; output style, format, tone, tool-call conventions → fine-tuning; a genuinely new capability or language → continued pretraining. If the scenario says 'updated weekly' or 'must cite the source', retrieval is the answer.
Computes GPU memory for serving as parameters × bytes per parameter only, and concludes a 70B model in FP16 'needs about 140 GB', ignoring the KV cache and activation/workspace overhead.
Training-time intuition dominates: during training the optimizer states dominate, so the KV cache is invisible. Candidates also forget that KV cache grows linearly with batch × sequence length and is allocated per layer.
What to do instead
Use the explicit budget: weights (params × bytes) + KV cache (2 × layers × kv_heads × head_dim × seq_len × batch × bytes_per_element) + activations/workspace + framework overhead (~10-20%). Work one concrete example end to end so the KV term is never dropped.
Asserts that INT4/FP8 weight-only quantization reduces KV cache memory and therefore raises the maximum batch size proportionally, or that quantizing weights makes decode compute-bound.
All 'quantization' is treated as one undifferentiated lever. The candidate does not distinguish weight memory, activation memory, KV-cache dtype, and arithmetic throughput, and does not recall that autoregressive decode is memory-bandwidth-bound.
What to do instead
Separate the three axes explicitly: weight quantization shrinks parameter memory and helps memory-bound decode; KV-cache quantization (e.g., FP8 KV cache) is a separate setting that shrinks the cache; compute speedups require quantized matmul kernels on supported hardware. Check which axis the question is actually about.
Budgets context windows and chunk sizes in words or characters - e.g., 'a 4096-token window holds roughly 4096 words' - or chunks a corpus by fixed character count with no tokenizer check.
Word/character heuristics are repeated in blog posts and feel safe, and candidates rarely tokenize sample text to see the real ratio, which varies by language, code, and formatting.
What to do instead
Always convert with the actual tokenizer for the model in question and state the measured ratio from a sample. When chunking, set chunk size and overlap in tokens, then verify the chunk-plus-prompt total fits the budget after reserving output tokens.
Evaluates a classifier or generator on an imbalanced or open-ended task with a single aggregate number (accuracy, or BLEU/ROUGE alone) and declares the system ready without per-slice breakdown.
Aggregate metrics are the default in tutorials and are easy to compute; the candidate does not ask what decision the metric will drive, and does not consider that a majority-class baseline can score 95% accuracy.
What to do instead
Choose the metric from the error cost (precision/recall/F1 or PR-AUC under imbalance; task-specific correctness plus human review for generation), always report the trivial baseline, and break results down by slice (language, length, document type, tenant) before concluding.
Splits data randomly into train/eval when the corpus contains near-duplicate documents, templated boilerplate, or the same entity appearing in both splits, then reports an inflated evaluation score.
Random splitting is treated as neutral. The candidate does not think about how the corpus was collected, so paraphrases and mirrors of the same source leak across the split boundary.
What to do instead
Deduplicate before splitting using MinHash/SimHash or n-gram overlap, and split by group (document source, user, or time) rather than by row. State explicitly in the answer that contamination control is part of the eval design, not an afterthought.
Reports a single run of a prompt or fine-tune as a definitive improvement over the previous version, with no baseline, no fixed seed, and no variance estimate, when the observed gain is a few points.
A/B comparison feels rigorous, and generation outputs look deterministic enough that sampling noise is underestimated. Temperature, sampling, and data ordering are not treated as sources of variance.
What to do instead
Register the comparison in advance: define the baseline (zero-shot or the current production prompt), fix seeds and decoding parameters, run multiple seeds or bootstrap resamples, and report a confidence interval. Claim an improvement only when the interval excludes the baseline.
Designs a RAG pipeline as embed → top-k → stuff into prompt, with no reranking, no chunk overlap or metadata filtering, and no requirement that the answer cite the retrieved passage.
The simplest tutorial pipeline is memorized as 'the RAG architecture', so retrieval quality is assumed rather than engineered, and the candidate never asks what happens when top-k returns near-miss chunks.
What to do instead
State the stages explicitly: parse and chunk with overlap and metadata, hybrid (dense + lexical) retrieval, rerank the candidate set, then assemble a prompt with numbered sources and require inline citations. Add a retrieval-quality check (recall@k on labeled queries) separate from the generation check.
Writes code that parses the model's free-text output with a naive split or json.loads and no schema validation, no retry path, and no timeout/retry policy on the API call - then treats a downstream crash as a model quality problem.
Prototype code that worked once on a friendly prompt is treated as production-ready. Failure modes of stochastic output and of network calls are not part of the mental model of 'calling an LLM'.
What to do instead
Constrain output with a schema (JSON mode / function calling / grammar), validate with a typed parser, and on failure re-prompt with the validation error included, up to a bounded retry count. Wrap calls with timeouts, exponential backoff on retryable errors, and a deterministic fallback path.
Treats prompt injection and jailbreaks as solved by a system prompt that says 'ignore malicious instructions', or ships an LLM-as-judge safety filter without calibrating it against labeled examples.
Guardrails are associated with prompt wording rather than with an evaluated defense-in-depth stack. The judge model's own error rate is assumed to be near zero because it is 'the same model family'.
What to do instead
Describe layered controls: input/output filtering, least-privilege tool access, treating retrieved text as untrusted data with delimiters, and human escalation for high-impact actions. Validate any automated judge against a labeled red-team set and report its precision/recall, including false-refusal rate on benign prompts.