Continuous Improvement of Test Automation
Continuous Improvement of Test Automation
This domain examines how to keep automation valuable after initial delivery. The exam tests maintenance planning, technical debt, retrospectives, and the use of metrics to improve the suite and its process.
Core rules
- Maintenance is a planned activity: corrective fixes, adaptive changes for new environments, perfective improvements, and preventive refactoring.
- Technical debt in test code is real debt; duplicated logic, brittle locators, and hardcoded data raise future cost.
- Use metrics and retrospectives to prioritize improvements, but avoid optimizing a number that does not reflect risk or value.
- Quarantine flaky tests, fix root causes, and track recurrence rather than repeatedly rerunning failures.
- Keep tool versions, dependencies, environments, and test data aligned with the system under test.
- Maintain an improvement backlog with owners and review it regularly.
Worked example
A suite shows 15 percent flaky tests. The team quarantines them, records causes, and finds that most failures come from fixed sleeps and shared test data. They replace sleeps with explicit waits and isolate data, reducing flakiness to 3 percent over two sprints. Reasoning: the metric identified the problem, root-cause analysis directed the fix, and the follow-up metric proved improvement.
Traps
- Deferring maintenance until the suite is unusable.
- Rerunning flaky tests without fixing causes.
- Upgrading tools without regression-checking the suite.
- Improving a metric without linking it to risk or cost.