Automated Testing Strategy in the Pipeline
The exam tests whether you can choose the right test level for each risk and keep the suite deterministic. Strategy questions usually present a slow, flaky suite and ask what to change.
What each level proves
- Unit: logic in isolation; milliseconds, deterministic.
- Component/service: one service with dependencies stubbed or containerised.
- Contract (consumer-driven, e.g. Pact): both sides of an interface agree without a full end-to-end run.
- Integration: real database, queue, network boundary.
- E2E/UI: a thin slice of critical user journeys; slow and brittle.
- Exploratory: finds unknown unknowns and usability defects that scripts cannot.
The pyramid (or trophy) shape follows: many fast low-level tests, very few E2E. In a microservice estate, contract tests replace most cross-service E2E because they localise the failure instead of leaving you with 40 services to debug.
Determinism
Tests must be hermetic: own data, own namespace, no shared mutable fixtures, no dependence on wall clock or execution order, seeded randomness. Quarantine flaky tests and fix them; silent retries hide real nondeterminism and inflate confidence.
Test data
Generate per run via factories, or provision isolated schemas. Never copy production personal data into test environments.
Risk-based selection
Run fast suites on every commit, longer suites on merge to trunk or nightly, and select a targeted subset based on changed code paths.
Worked example
A 40-service estate runs 500 E2E tests taking 4 h; about 60% of failures are environment flakes. Introducing consumer-driven contract tests for each service pair gives roughly 120 fast tests in 6 min, covering interface integration risk. Keep 40 E2E smoke journeys on the critical user path and delete the rest. Integration coverage of the risky boundary remains while feedback drops from hours to minutes.
Traps that cost marks
- Using coverage percentage as a quality target; it measures execution, not assertion.
- Replacing all E2E with unit tests, so nothing verifies real wiring.
- Retrying flaky tests indefinitely until they pass.
- Shared test databases where pass/fail depends on run order.
- Testing against a production-like environment only after merge, so every defect costs a full pipeline cycle.