Generative AI Fundamentals
Domain 1 tests whether you can explain, in mechanism terms, how generative AI produces an answer - and therefore why it behaves as it does. Expect vocabulary questions on tokens, embeddings, transformers, training stages and sampling settings, plus scenario questions that ask you to predict model behaviour.
Generative versus discriminative
A discriminative model labels input (spam / not spam). A generative model learns the distribution of the data and produces new samples from it. Large language models do this by predicting the next token; diffusion models iteratively denoise random noise into an image. Both are trained on large corpora and then aligned to human instructions.
The four stages
- Pre-training: self-supervised next-token prediction over huge text corpora. This creates a foundation model whose 'knowledge' is statistical and frozen at a cut-off date.
- Alignment: instruction tuning and preference tuning (for example RLHF) shape the model to follow requests and preferred style.
- Inference: the model generates tokens one at a time inside a context window. Cost and limits are measured in tokens, not words.
- Grounding: retrieval-augmented generation (RAG) and tools supply fresh or private data at inference. RAG changes the prompt, not the weights.
Tokens, embeddings, parameters
A tokenizer splits text into subword tokens, so a context window of N tokens is roughly three-quarters of that in English words. Embeddings are vectors representing meaning; they power semantic search in RAG. Parameters are the learned weights - they are not tokens, and prompts do not update them.
Sampling controls
Temperature, top-p and top-k control randomness. Temperature near zero selects the highest-probability token each step: more repeatable, but not guaranteed identical across model versions.
Worked example
You ask the same extraction question three times and get three slightly different answers. Reasoning: the default temperature is above zero, so each token is sampled from a probability distribution; small variations compound into different phrasing and occasionally different extracted values. The fix is to lower temperature, constrain output to a JSON schema with an explicit null-if-absent rule, and validate programmatically. Lower temperature reduces variance, not error - deterministic output can still be wrong.
Traps that cost marks
- Saying the model 'looks up' facts; it predicts tokens, and retrieval is an add-on.
- Confusing parameters (weights) with tokens (units of text and billing).
- Claiming a bigger context window guarantees the model uses every part of it.
- Assuming fine-tuning is the right way to inject frequently changing facts; RAG usually is.