Deterministic test data and a virtualised payment switch under data protection constraints
MediClaim processes healthcare claims and is subject to data protection rules. Test environments use a masked copy of production refreshed every eight weeks. Between refreshes, testers create ad-hoc accounts, and automation scripts reference hard-coded member IDs that frequently disappear or change status after a refresh. Roughly 9% of the 700 API tests fail at data setup, and another 6% fail because the downstream payment switch is only available in production. Stub coverage is patchy: teams wrote three different mocks of the same switch with different behaviour, none validated against the real interface. Compliance has also flagged that the masked copy retains combinations of birth date and postcode that could re-identify individuals. The programme wants deterministic, repeatable tests without abandoning realism, and has to demonstrate that test data handling is defensible at audit. A payment switch contract (OpenAPI specification) is available and versioned.
1. Recommend a test data architecture. Address freshness, determinism, realism and the audit requirement, and say what you would stop doing.
Read the worked answer
The failure pattern is architectural: tests depend on a periodic snapshot whose contents they do not control, so both determinism and data protection are compromised at once.
Stop treating the masked copy as the primary data source. It is non-deterministic by construction, drifts between refreshes, and masking does not remove re-identification risk where quasi-identifiers combine, which is exactly compliance's finding. Keep, if needed, only a small, tightly masked dataset for exploratory realism and investigation.
Introduce a data provisioning service. Tests request the data they need through an API, receiving freshly created records with unique identifiers, and release them at teardown with a time-to-live so orphans are reaped. Scripts stop referencing hard-coded member IDs and instead resolve identifiers at run time from the provisioning response. This removes the 9% setup failure class almost entirely and makes tests independently runnable in parallel.
Synthetic-first with constraint-aware generation. Generate members and claims from a model that respects domain rules (valid claim-to-member relationships, status transitions, date ordering) and is seeded, so the same seed reproduces the same dataset in any environment. Synthetic data is inherently defensible at audit: no personal data is processed, and the generation rules and seeds are documented artefacts.
Keep a thin realism layer. A defined set of regression scenarios should run against a masked, compliance-approved subset, with documented masking rules and a re-identification risk assessment, so the team can show both realism and control.
Governance. Version the generator and masks in source control, record data lineage for audit, define who may create production-derived data, and add a check that no test script contains a literal member identifier.
2. How would you handle the payment switch: three inconsistent mocks, a versioned OpenAPI contract, and 6% of tests failing because the real system is unavailable?
Read the worked answer
Three divergent mocks of one interface is a correctness hazard: tests assert behaviour that may not exist in production, and results differ by which mock an environment happens to load. The versioned contract is the anchor for fixing this.
Consolidate to one service virtualisation asset, generated from and validated against the contract. Replace the three hand-written mocks with a single stub implementation whose request and response schemas are checked against the OpenAPI specification in the pipeline, so schema drift fails the build. Run a contract check on every contract version change, and add schema validation of stub responses as a gate.
Model behaviour, not just responses. The stub must support the stateful and exceptional paths the tests need: authorisation approved, declined, timeout, duplicate submission, partial settlement. Drive these from data or scenario names in the test, not from per-team code forks. Define the specific non-deterministic behaviours the tests depend on (response latency, out-of-order callbacks) explicitly, so tests that need them get them deterministically.
Keep a real-interface lane. Stubs encode assumptions. Run a small, high-value set of end-to-end tests against the real switch in a nightly or pre-release lane, and reconcile any divergence as a defect in the stub or the contract. Log stub-versus-real response differences where the real system is reachable at all.
Ownership and versioning. Assign one owner for the virtual asset, version it alongside the contract, and publish which version each environment uses. Three mocks persisted because nobody owned the interface; consolidation without ownership will regress.
3. What risks does heavy reliance on synthetic data and stubs introduce, and how would you keep them visible and controlled?
Read the worked answer
Synthetic data and stubs buy determinism at the price of fidelity, and unmanaged fidelity loss produces a suite that is green while production breaks.
Synthetic data risks. Generators encode the team's assumptions about valid data; real data contains legacy anomalies, encoding quirks, missing fields and historical edge cases that the generator will not invent unless someone thought of them. There is also a coverage risk: the generator may produce a comfortable, narrow distribution, so tests never see genuine boundary cases. Control: derive generation rules from production profiling (distributions, null rates, format variants, not actual values), review the rules with business and data owners, and deliberately seed known hard cases into the generator. Compare a sample of generated records against production profiles each quarter.
Stub risks. A stub asserts the team's belief about the payment switch, including timing, error codes and idempotency behaviour. If that belief is wrong, tests pass and production fails. Control: validate against the versioned contract, run the real-interface lane nightly, and treat any stub-versus-real divergence as a defect with an owner and a fix date. Track how long each stub has gone without real-interface reconciliation.
Governance that keeps both visible. Publish a fidelity register listing each stub and synthetic dataset, its assumptions, owner, last validation date against the real interface or production profile, and known gaps. Review it at release readiness, and require a real end-to-end pass, however thin, before any release that touches the payment path. Add a def:done rule that new stubs cannot be introduced without contract validation and an entry in the register.