Continuous Testing Foundation cheat sheet

Continuous testing is the practice of executing automated tests as part of the software delivery pipeline to obtain immediate feedback on business risks; it is not a phase but a continuous activity.

Shift-left testing moves test activities earlier in the lifecycle (requirements, design, coding); shift-right testing includes production monitoring, canary analysis, and chaos engineering.

Test pyramid (Cohn): many unit tests, fewer service/API tests, very few UI tests; the anti-pattern is the ice-cream cone (many manual/UI tests, few unit tests).

Testing trophy (Dodds): emphasizes integration tests as the largest layer, with static analysis at the base, unit tests next, and E2E at the top; suited for modern front-end and API-heavy systems.

Key metrics: test execution time, flakiness rate, defect escape rate, mean time to detect (MTTD), mean time to recover (MTTR), and test coverage (statement, branch, path).

Benefits of continuous testing include faster feedback, lower cost of defect removal (defects found earlier cost less), reduced risk, and higher release confidence.

Terms: test oracle (mechanism to determine pass/fail), test double (stub, mock, fake, spy), fixture (precondition state), and hermetic test (isolated, repeatable environment).

Continuous testing in DevOps is enabled by test automation, service virtualization, test data management, and environment provisioning as code.

Culture of quality: testing is a whole-team responsibility, not a separate QA silo; developers write and maintain automated tests.

Blameless postmortems focus on system and process improvement, not individual punishment; they feed test coverage and monitoring improvements.

Definition of Done (DoD) should include automated tests passing, code reviewed, and quality gates met before a story is considered complete.

Collaboration practices: three amigos (business, development, testing) refine acceptance criteria and examples before coding; behavior-driven development (BDD) uses Given-When-Then.

Continuous feedback loops: fast pipeline feedback, visible test results, and shared dashboards reduce handoffs and waiting time.

Psychological safety encourages reporting defects and test failures early; metrics should not be used to blame individuals.

Cross-functional teams own test strategy, test data, and environment needs; QA coaches rather than gatekeeps.

Leadership support: allocate time for test automation, technical debt reduction, and training; recognize quality as a shared goal.

Test early and often: run unit tests on every commit, integration/API tests on every merge, and end-to-end/regression tests on every deployment to staging.

Risk-based testing prioritizes test effort by likelihood and impact of failure; high-risk areas get more automation and exploratory testing.

Test segmentation: smoke tests (fast, critical path) gate deployments; full regression runs asynchronously; performance and security tests run on a schedule or pre-release.

Shift-right strategies include canary releases, blue-green deployments, feature flags, synthetic monitoring, and chaos experiments in production.

Service virtualization and contract testing (e.g., consumer-driven contracts) allow testing without full dependent environments.

Parallelization and test distribution reduce feedback time; use test impact analysis to select relevant tests for each change.

Environment strategy: ephemeral environments provisioned via infrastructure as code; test data is synthetic, masked, or subsetted to protect privacy.

Flaky test management: quarantine flaky tests, fix or remove them, and track flakiness rate as a quality metric.

Unit test frameworks: JUnit (Java), pytest (Python), NUnit/xUnit (C#), Jest/Mocha (JavaScript); they integrate with build tools and CI servers.

BDD/ATDD tools: Cucumber, SpecFlow, Behave; they map plain-language scenarios to step definitions and promote collaboration.

API testing tools: Postman, REST Assured, SoapUI, Karate; they validate contracts, status codes, and payloads.

UI automation: Selenium WebDriver, Cypress, Playwright; use page object model and resilient selectors to reduce flakiness.

CI/CD tools: Jenkins, GitLab CI, GitHub Actions, Azure Pipelines, CircleCI; they orchestrate test stages and publish reports.

Test management and reporting: Allure, JUnit XML reports, TestRail, Zephyr; integrate results into dashboards and quality gates.

Service virtualization: WireMock, Mountebank, Hoverfly; contract testing: Pact, Spring Cloud Contract.

Performance testing: JMeter, Gatling, k6, Locust; security testing: OWASP ZAP, SonarQube (static analysis), dependency scanners.

Test plan should be lightweight and living: define scope, risks, test levels, environments, data, entry/exit criteria, and metrics; review regularly.

Define quality gates: e.g., no critical defects open, unit test coverage threshold, all smoke tests pass, security scan clean.

Use risk assessment matrix (probability × impact) to prioritize test cases and automation candidates.

Plan test data and environments early; include data refresh, masking, and provisioning in the plan.

Include non-functional testing (performance, security, accessibility, resilience) in the plan from the start.

Entry criteria: build deployed, test environment ready, test data available; exit criteria: planned tests executed, defects triaged, quality gates met.

Plan for continuous testing in the pipeline: define which tests run at which stage and the maximum acceptable feedback time.

Traceability: link tests to requirements/user stories to ensure coverage and support impact analysis.

Automate the right tests: stable, repeatable, high-value tests first; avoid automating unstable or one-off scenarios.

Follow the test pyramid: write many fast unit tests, fewer integration/API tests, and minimal UI tests.

Design for testability: dependency injection, interfaces, and separation of concerns make code easier to test.

Use the page object model and behavior-driven selectors for UI tests; avoid brittle XPath tied to layout.

Make tests independent, idempotent, and hermetic; each test sets up and tears down its own data.

Use test doubles (mocks, stubs) for external dependencies; contract tests for service boundaries.

Implement continuous integration: run tests on every commit, fail fast, and keep the build green.

Measure automation ROI: consider maintenance cost, execution time, and defect detection effectiveness; retire low-value tests.

Centralize test results and metrics in dashboards; use trend analysis rather than single-run pass/fail.

Track key metrics: test execution time, pass/fail rate, flakiness rate, defect density, defect escape rate, and coverage.

Use test impact analysis to select and prioritize tests based on code changes, reducing feedback time.

Classify defects by root cause (requirements, design, code, test, environment) to improve process.

Manage test data and environments as assets: version, refresh, and secure them.

Conduct regular test reviews and retrospectives to improve test effectiveness and efficiency.

Integrate test management with ALM/issue trackers for traceability and reporting.

Use risk-based reporting: highlight high-risk areas and unresolved defects to stakeholders.

A test strategy defines the overall approach: test levels, types, environments, tools, metrics, and quality gates aligned with business goals.

Align test strategy with DevOps principles: automation, continuous feedback, shift-left/right, and collaboration.

Define test levels: unit, component, integration, system, acceptance; and test types: functional, performance, security, usability, accessibility.

Specify quality gates and exit criteria for each pipeline stage (commit, build, deploy to staging, production release).

Select tools and frameworks that integrate with the CI/CD pipeline and support reporting and traceability.

Define roles and responsibilities: developers own unit tests, QA owns test strategy and exploratory testing, operations owns monitoring.

Include a roadmap for test automation, environment provisioning, and test data management.

Establish metrics and continuous improvement: review test strategy regularly based on feedback and incidents.