LLM-Generated API Tests in a Payments Migration
PayBridge is migrating 24 REST endpoints from a monolith to microservices. A test engineer with one year of automation experience asks an LLM to generate a pytest suite from the OpenAPI 3.0 specification. In one day it produces 180 tests. The first CI run has 72 failures: 41 use hardcoded account IDs and fixed 2023 dates; 18 assert only HTTP 200; 9 call a deprecated endpoint; 4 expose genuine defects in refund calculation. The suite takes 55 minutes and fails intermittently because tests share seeded accounts. The engineer proposes making the suite a mandatory release gate and enabling the LLM's self-healing mode to rewrite selectors and assertions automatically. The product owner likes the speed; the QA lead is worried about false confidence. There is a nightly environment with a payment simulator, a contract-testing tool, and two reviewers available.
1. Classify the 72 failures and decide whether the suite should become a mandatory release gate now.
Read the worked answer
Classification. The 41 hardcoded-data failures are test-design defects, not product failures. The 18 status-only assertions are weak oracles that can pass while business behavior is wrong. The 9 deprecated-endpoint calls indicate specification drift or missing version control. Only the 4 refund-calculation failures are credible product defects and should be triaged immediately.
Decision. Do not make the suite a mandatory gate now. A gate that fails mostly for generated-test defects blocks releases for the wrong reason and trains teams to ignore red results. It also creates false confidence because weak assertions can pass. First separate the 4 product defects, fix or quarantine the 68 non-product failures, and establish a stable baseline. Gate only after the suite meets agreed stability, oracle-strength, and runtime criteria. The product owner’s speed argument is valid for idea generation, not for release control.
2. Propose remediation for test data, assertions, and API contract coverage.
Read the worked answer
Test data: replace hardcoded IDs and dates with factories that create unique accounts per test, seed them deterministically, and clean up afterward. Isolate data for parallel runs and avoid shared accounts. Use the payment simulator for external dependencies and reset state between tests.
Assertions: require schema validation, business invariants, idempotency checks, error codes, and negative cases, not just HTTP 200. For refunds, assert calculated amounts, ledger entries, and state transitions. Add contract tests against the OpenAPI specification and version the spec so deprecated endpoints fail visibly. Keep a small number of end-to-end tests and push validation into API and component tests to control the 55-minute runtime.
Review: have the two reviewers check generated tests for secrets, unsafe data, and weak assertions before merge. Static analysis and linting should run in CI.
3. What metrics and human-review rules should govern generated test code, and how should self-healing be controlled?
Read the worked answer
Track false-positive rate, escaped defects, requirement or endpoint coverage, mutation score where feasible, flake rate, runtime, and reviewer acceptance rate. Also measure how often generated tests are modified before merge; high modification rates signal prompt or specification problems.
Human review is mandatory for new or changed assertions, authentication handling, and destructive operations. Self-healing can be useful for selectors in UI tests, but in API tests it can rewrite assertions and hide a real contract change. Require self-healing changes to be surfaced as diffs, reviewed, and never allowed to weaken assertions or auto-approve a release. Alert when self-healing activates repeatedly for the same test, because that indicates instability or drift. Adopt the gate only after false positives and flake rate fall below agreed thresholds.