Applying Generative AI to Test Implementation, Automation and Execution
What the domain tests
This domain covers GenAI for test implementation, automation and execution: generating or refactoring test code, locators, test data, mocks, and assisting failure triage. The exam tests whether you can use generated code safely in a pipeline.
Core rules
- Generated code is a proposal. It must compile, run, assert meaningful outcomes, clean up state and follow project standards.
- Review for waits and synchronisation, isolation, data setup, error handling and security. Prefer stable locators (role, label, test id) over brittle XPath; validate every locator against the DOM.
- Test data: use synthetic data with realistic constraints; never put production PII or secrets in prompts; label synthetic data and check uniqueness and referential integrity.
- Execution support: use models to cluster failures, summarize logs and suggest causes, but keep the evidence link. Triage suggestions are hypotheses, not verdicts.
- Version prompts, pin model versions and regression-test prompt changes; require human approval before auto-committing generated tests.
Worked example
A tester asks for a Playwright login test. The model returns a test using a long absolute XPath, a fixed sleep, and no assertion for the error message. The reviewer replaces the locator with getByLabel, removes the sleep in favour of web-first assertions, adds a negative case for invalid credentials, and parameterises users. The test then runs in CI. Reasoning: the model accelerated boilerplate, but correctness, maintainability and oracle quality came from review.
Traps
Trusting generated code because it looks idiomatic; hard-coded sleeps; secrets in prompts; self-healing that masks real defects; unreviewed auto-commits; treating clustered failures as confirmed root causes.