Test Tools, Automation, and Metrics
What this domain tests
This domain tests whether you can choose tools and metrics that improve coverage and feedback speed rather than create maintenance drag.
Core rules
- Tool categories: test management and traceability, defect tracking, automation frameworks, CI and CD, device farms, crash reporting, telemetry, performance profilers, and network simulators.
- Automate stable, repetitive, high-value checks first: smoke tests, build verification, save and load regression, and performance benchmarks.
- Do not automate unstable UI or one-off exploratory checks. Flaky tests erode trust in the whole suite.
- Metrics include pass rate, defect density, defect escape rate, requirement and code coverage, mean time to detect and fix, flaky rate, and test execution time.
- Pair every metric with a decision. Vanity metrics without an action are noise.
Worked example
A 30-minute smoke suite runs on every CI build and catches 20% of blockers before QA. Expected return on investment is faster feedback. Actual: 40% of runs fail for timing reasons. The reasoning: stabilize waits and isolate test data before expanding automation. A flaky suite is worse than no suite because it trains people to ignore red builds.
Traps
Automating everything; measuring test case count instead of coverage; ignoring maintenance cost; using pass rate alone when tests are weak; lacking traceability from requirement to test to defect.