Verifying and Validating Test Automation
Verifying and Validating Test Automation
This domain tests whether you can prove that the test automation solution is built correctly and that it does the right job. Verification checks the solution against its requirements and design; validation checks whether it detects the defects it is meant to detect.
Core rules
- Verification activities include reviews, static analysis, unit tests for framework code, and checks against automation requirements.
- Validation requires evidence that tests can fail: inject known defects or use mutation techniques, and confirm the suite detects them.
- Also run the suite against a correct system to detect false positives.
- Quality characteristics for automation include reliability, maintainability, usability, performance, portability, and security.
- Flaky results are a validation failure until proven otherwise; they undermine trust and must be root-caused.
Worked example
A suite passes on the current build. The engineer injects a known pricing defect into a test environment; the relevant check still passes. Reasoning: the suite has a false negative, so it is not validated even though it is “green.” The team reviews the assertion and data setup, fixes the check, and re-injects the defect to confirm detection.
Traps
- Treating a successful run as proof of detection ability.
- Testing only the happy path of the automation code.
- Ignoring false positives that waste investigation time.
- Accepting intermittent failures as normal.