ISTQB CT-GenAI common mistakes
ISTQB CT-GenAI · Understand the traps and how to avoid them.
Treating the LLM as a deterministic oracle and asserting that a generated test must be 'correct' because the model produced it, rather than classifying outputs as probabilistic suggestions requiring human verification.
Candidates carry over habits from conventional tooling, where an API returns the same answer every time. Generative AI is non-deterministic (temperature, sampling), so the same prompt yields different outputs, and the exam tests whether you understand that verification, not trust, is the default posture.
What to do instead
Before answering any scenario question, ask 'who verifies this output and how?' If the option removes human review or assumes reproducibility, eliminate it. Memorise the framing: generative AI produces candidate artifacts; the tester owns correctness, and oracles must be defined independently of the model.
Confusing hallucination with other defect types, e.g. labelling a plausible but fabricated API method as 'data leakage' or 'prompt injection', or calling a stale-but-true statement a hallucination.
The failure categories (hallucination, outdated knowledge, bias, prompt injection, data leakage, non-determinism) are learned as a list of words rather than as distinguishable mechanisms, and exam distractors are deliberately adjacent.
What to do instead
Build a one-line discriminator for each: hallucination = fluent output not grounded in fact or source; outdated knowledge = training cut-off; prompt injection = untrusted input altering instructions; leakage = sensitive data surfacing in output. Practise mapping short scenarios to exactly one category and justify the exclusion of the runner-up.
Writing prompts that ask for 'good test cases for the login page' and expecting usable output, then blaming the model when results are generic.
Candidates underestimate prompt engineering as a testable skill. The exam rewards specificity: role, context, input format, constraints, output schema, examples, and explicit assumptions. Vague prompts are the most common root cause of poor generated artifacts.
What to do instead
Adopt a checklist for every prompt-design question: objective, context and domain constraints, input data or specification excerpt, required output format, quality criteria, and instruction to state assumptions or ask for missing information. When an option adds context or a format constraint, prefer it over one that merely rewords the request.
Accepting generated test conditions without checking coverage against the source specification, so requirement gaps and duplicated conditions go unnoticed.
Generated suites look comprehensive because they are verbose. Candidates equate volume with coverage and skip the traceability step that the syllabus places at the centre of AI-assisted analysis and design.
What to do instead
Always pair AI-generated conditions with an explicit traceability check against requirements, risks, and existing coverage. In scenario questions, choose the option that adds a coverage or gap analysis step, and reject options that treat generated volume as evidence of completeness.
Using generative AI output as the expected-result oracle (e.g. asking the model what the API should return and asserting that), instead of deriving expected results from the specification or a trusted source.
It is convenient and fast, and the model's answers often look right. But an oracle derived from the same probabilistic system under test's domain knowledge can mask real defects, producing tests that pass while the product is wrong.
What to do instead
For any question about expected results, insist the oracle comes from requirements, models, or independent calculation. Treat model-generated expected values as drafts to be validated. If an answer option makes the AI the sole source of truth for pass/fail, eliminate it.
Pasting production data, credentials, or customer records into a public generative AI tool to generate test data, without anonymisation or an approved environment.
Candidates focus on the technical task and treat the tool as a neutral editor. Confidentiality, licensing, and data-protection obligations are exam-relevant constraints, and this is a frequent scenario stem.
What to do instead
Default to synthetic or masked data, check the tool's data-retention and training-use policy, and prefer enterprise or on-premises deployments for sensitive inputs. When an option includes anonymisation, access control, or policy review before prompting, it is usually the intended answer.
Assuming AI-generated automation code is ready to merge, ignoring that locators, waits, error handling, and assertions frequently need rework and that generated code must pass review and the existing quality gates.
The code compiles and reads plausibly, so candidates skip the review and stabilisation step. The exam tests the full implementation lifecycle, not just code generation.
What to do instead
Treat generated scripts as pull requests from a fast but inexperienced contributor: review, refactor, run against the real application, harden synchronisation, and verify assertions independently. Choose options that include peer review and execution feedback over options that skip straight to CI.
Describing AI-assisted test execution as if the model executes tests or makes pass/fail decisions autonomously, rather than supporting triage, log summarisation, and failure clustering under human control.
Marketing language about 'autonomous testing agents' blurs the boundary between the execution engine (runner, framework, CI) and the model's supportive role. Candidates attribute capabilities to the wrong component.
What to do instead
Keep the architecture explicit: the test runner executes, the model analyses artifacts (logs, stack traces, screenshots) and proposes hypotheses. For self-healing questions, remember that healing must be logged, reviewed, and must not silently change intended behaviour.
Producing AI-drafted defect reports and status summaries without verifying facts, deduplicating against existing defects, or separating observed evidence from model speculation.
Fluent summaries read as authoritative and save time, so the verification step is skipped. The exam checks whether you can keep reported metrics and defects traceable and free of invented detail.
What to do instead
Apply a rule: every claim in a report must trace to an artifact (log line, screenshot, test run ID). Verify, deduplicate, and label AI-drafted content as draft pending review. Reject options that publish generated summaries automatically to stakeholders.
Treating AI-assisted testing as a one-off productivity boost rather than a feedback loop, i.e. ignoring the need to measure whether generated tests find defects, track false positives, and refine prompts and guardrails over time.
Continuous improvement is a distinct domain and candidates under-prepare for it, defaulting to generic 'monitor and improve' answers instead of naming concrete metrics and feedback mechanisms.
What to do instead
Learn a small metric set: defect detection rate of generated tests, false-positive and flake rates, review rejection rate of generated code, coverage delta, and prompt-version outcomes. In questions about maturing the practice, prefer options that close the loop with measured data and updated prompts or guardrails over options that merely adopt more tools.