Fundamentals of Generative AI for Testing
What the domain tests
The Fundamentals domain checks whether you understand how generative AI produces test artifacts and why that changes test strategy. Generative models, especially large language models (LLMs), predict tokens probabilistically; they do not retrieve a stored answer. Output therefore varies with prompt, context, model version, temperature, top-p and max-token settings. The exam expects you to distinguish prompting, retrieval-augmented generation (RAG) and fine-tuning, and to explain grounding, embeddings and context-window limits.
Core rules
- Treat every output as a draft artifact requiring an oracle and human review.
- Ground prompts in the test basis: requirements, user stories, designs, logs. RAG supplies external context at inference time; fine-tuning changes model weights; prompting alone cannot guarantee facts.
- Lower temperature reduces variation but does not eliminate hallucination.
- Assess privacy, IP, security and bias before sending data to a model; prompts can leak secrets and prompt injection can override instructions.
- Select models on task fit, cost, latency, data residency and evaluation results, not reputation.
Worked example
A team asks an LLM to derive test conditions for a password-reset feature. A weak prompt says: 'List tests for password reset.' A stronger prompt supplies the requirement excerpt, specifies role, output columns (condition, source, risk, oracle), asks for assumptions, and sets temperature 0.2. The model returns conditions including expired token and reused token, but invents a 15-minute expiry. The reviewer traces every condition to the requirement, deletes the invented value, adds a missing condition for rate limiting, and records the prompt/model version. Reasoning: grounding improves relevance, structured output improves review, and traceability exposes hallucination.
Traps
Assuming deterministic output; treating fluent text as correct; confusing RAG with fine-tuning; exceeding the context window and losing instructions; sending PII or credentials in prompts; skipping human review because the model is 'smart'.