Fitting 400 executions into a 4,860-case generated suite

Selection must be driven by the model structure and the risk register, not by sampling.

Step 1 - state-machine behaviour first (about 200 cases). The offline-queue, retry and fallback region carries the highest severity and two recent severity-1 defects. Cover it at 1-switch (transition-pair) level and add explicit retry-limit boundary cases (attempt 1, limit−1, limit, limit+1, and the fallback after exhaustion). If that needs 200 cases, it gets 200; this is the component where a miss is a certification failure, not a delay.

Step 2 - parameter combinations (about 150 cases). Use pairwise (2-way) coverage of the five parameters, which typically costs a couple of dozen cases, then force in: all values of card scheme, all currency exponents, and any known boundary values. Pairwise is chosen because most failures are triggered by one or two parameters, and 3-way would exceed the budget for little risk reduction.

Step 3 - regression insurance (about 50 cases). Every test case that previously found a defect and every case touching settlement/batching, regardless of coverage contribution.

Justify the split to stakeholders as risk-weighted, and report which coverage level each subset achieves so the 400 is auditable.

Before generation, in the model. Encode the coupling as an invariant: profile 'P3' implies connectivity 'offline-capable'. The generator then never emits the impossible combination.

Why not filter after generation. Post-generation filtering has three costs. It wastes generation and triage effort on cases that were never valid. It destroys the audit trail - a filtered case looks identical to a case nobody thought of, so you cannot later prove the gap was known and justified. And it corrupts coverage arithmetic: coverage has to be measured against the feasible set, so you need the feasibility constraint in the model to state the denominator honestly.

Distinguish two kinds of infeasibility, because they are handled differently in reporting:

• Invalid data combinations (P3 with online) - a constraint on parameter values; excluded by an invariant.
• Unreachable paths in the state machine - e.g. a retry transition guarded by a condition that a prior transition has already made false. These need a feasibility analysis of the model and written justification, because a path that looks unreachable may indicate a modelling error rather than a genuine impossibility.

Report both as 'excluded with reason' so the coverage figure remains falsifiable.

Stopping criteria (all must hold, not just the budget):

• 1-switch coverage of the offline-queue/retry region achieved, with every exclusion documented.
• All retry-limit and timeout boundaries exercised.
• Pairwise coverage of the feasible parameter space at 100%, with the forced high-risk values included.
• Defect discovery rate flat over the last ~60 executions - if new defects are still arriving, the 400 figure is an arbitrary cut, not a sufficiency argument.
• No unexplained failures outstanding at the end.

Reasons to distrust an all-green run:

• Bench instability. A shared bench with intermittent failures means some 'passes' are luck. Require deterministic setup or re-run of failures.
• Oracle weakness. If expected results come from a stale reference implementation, 400 passes prove only that two systems agree, not that either is right - check that expected values derive from the model, not from the previous build.
• Selection bias. If ad-hoc selection crept in, the 400 may be the 400 easiest cases. Verify the selection is reproducible from the model and the risk list.
• Coverage arithmetic. A 100% figure computed over a shrunk feasible set (silently dropping infeasible paths) is not the same claim.