6,912 Combinations, Four Testers, Nine Days
You are the test lead on a 12v12 hero shooter two weeks from a content milestone. The new build adds 12 weapons, 8 attachment types, 6 maps, 3 game modes and 4 supported input configurations (keyboard and mouse, two controller layouts, handheld). A naive matrix is 12 x 8 x 6 x 3 x 4 = 6,912 combinations. You have 4 testers and 9 working days, and the team has already burned two days on a full pass of one map. Design constraints: three weapons accept no attachments at all, two attachments are mutually exclusive, and attachments are disabled entirely in one mode. The combat designer says the highest-risk interaction is between the recoil-reduction attachment, the burst-fire weapon and the vertical map geometry, and the build has a known three-way stacking exploit. The publisher's milestone checklist requires documented coverage of every weapon and every mode.
1. Propose a combination strategy for the nine days, justify it, and state what you would still test exhaustively.
Read the worked answer
Strategy
- Constrain before you combine. Apply the rules first: three weapons take no attachments (9 effective weapons), two attachments are mutually exclusive, attachments are off in one mode. The raw 6,912 shrinks by roughly a third before any statistical trick is used.
- Partition by factor importance. Weapon and mode are high-risk and cheap (12 and 3 levels), so test all 36 with a default loadout exhaustively. All 6 maps get covered, but map only interacts with geometry-sensitive weapons (sniper, explosives, burst rifle) - targeted pairs rather than a full cross.
- Pairwise for the rest. Attachments × weapon × input config through a covering array (PICT or ACTS) gives roughly 30 to 40 cases for complete two-way coverage of valid combinations. Encode the constraints as model rules so invalid rows are excluded by construction, not filtered afterwards.
- Add the known high-order risks on top. Pairwise is a defect-detection heuristic, not a guarantee. The recoil-reduction × burst rifle × vertical geometry interaction is a known three-way risk, so enumerate that triple across maps and modes explicitly, plus the stacking exploit permutations.
Test exhaustively, regardless of time Every weapon fires, reloads and switches at least once per mode; every mode's win, lose and score paths; loadout persists across restart; every binding on all four input configs. These are cheap, high-signal and not negotiable - they are the cases whose failure is a headline.
2. How do you handle the designer's known three-way interaction when the plan relies on pairwise coverage?
Read the worked answer
Why pairwise does not cover it A covering array guarantees every pair of levels appears at least once. It guarantees nothing about triples. A known three-way interaction is therefore a structural blind spot: a clean pairwise run tells you nothing about it, and reporting the run as reassurance would be misleading.
What to do instead
- Treat it as a directed test: enumerate the triple (recoil-reduction attachment × burst rifle × each vertical map area) in full, or across a subset with a written rationale if mode rules make some rows invalid.
- Then ask whether the interaction generalises. Does the same stacking behaviour appear with other recoil-modifying attachments or other multi-shot weapons? If yes, the risk class is multiplicative modifiers applied per projectile, and you expand the class rather than the single instance. That is the difference between fixing a bug and fixing a family of bugs.
- Capture the designer's risk rating as the source in the test plan, so the coverage decision is documented and owned rather than looking like an arbitrary shortcut.
- Add the exact repro as a permanent regression case so the defect cannot silently return in a later content drop.
- If time runs out, the documented gap is what protects the team. An undocumented gap is what produces the day-one hotfix.
3. The publisher's checklist says all combinations must be covered. How do you report your coverage and defend the reduced approach?
Read the worked answer
Translate the requirement first Publishers usually want demonstrated coverage and a defensible method, not 6,912 executed rows. Ask what evidence they accept: a coverage matrix, an execution log, or a signed statement. Guessing wrong wastes a week.
Three artefacts to present
- Coverage matrix: every weapon × mode × map exercised at least once at P0, with pass status and build number, so the claim is checkable line by line.
- Method statement: constraints applied, the tool and version used, the exported covering array, and the explicitly enumerated high-risk triples with the reason each was chosen.
- Residual risk register: what pairwise cannot detect, which areas were sampled rather than exhausted, and any inconclusive results.
Show the arithmetic 6,912 raw, roughly 1,400 after constraints, about 180 executed for full two-way plus full P0 single-factor coverage. That is a stronger claim than running the full matrix, because it is verifiable and it names its own limits.
If they still insist Escalate with the cost: 6,912 rows at about 12 minutes each is roughly 1,380 tester-hours, which is infeasible at this milestone. Offer full enumeration of the top 10% risk-ranked combinations as a compromise.
The integrity point: never label a reduced suite as full coverage. The label is the ethical issue, not the reduction.