Ember Vault: Triage When 'Broken' and 'Intended' Collide
Bright Lantern Games is three weeks from launching Ember Vault, a single-player roguelike deckbuilder shipping simultaneously on PC and iOS. Six testers share a 20-device lab; there is no separate playtest team and no user-research budget left this quarter. Three items sit in the tracker. (1) A crash on launch occurs roughly once in 55 cold starts on iOS 15.4 devices and has never reproduced on iOS 16 or 17. (2) Beta telemetry from 4,000 players shows 62% stop playing at the second boss, 'The Hollow Choir', with a median of 11 attempts before quitting. (3) A tester reports that the card Mirror Shield reflects damage after the enemy death animation, so a player at zero health can still win the fight; the designer replies 'intended, it is a clutch mechanic'. The producer wants everything 'fixed' before launch and asks you to triage all three.
1. Classify items 1-3 and explain what each classification changes about ownership, tracking, and the release decision.
Read the worked answer
Item 1 is a defect: an observable failure to meet an implicit reliability expectation, with a reproduction rate (~1 in 55) and a bounded configuration (iOS 15.4). It belongs to engineering, is tracked with severity and priority, and blocks release for the affected OS share unless a documented risk acceptance is signed.
Item 2 is not a defect. Nothing has failed against a specification; it is a design/experience risk. Ownership sits with design and production, its evidence is distributional (attempts-to-clear, quit points), and the release decision is a product judgement, not a pass/fail gate. Filing it as a bug guarantees it will be closed as 'works as designed' and the signal lost.
Item 3 is a specification gap. The behaviour is implemented as designed, but the design is undocumented and its interaction with the death state is genuinely ambiguous. The correct action is to make the rule explicit, give it acceptance criteria, and test it - including the edge case where both players reach zero in the same frame. Classification matters because it routes the item to the person who can actually decide it and defines what evidence would close it.
2. For item 2, how would you design a playtest rather than a test? State what you would measure and what conclusion the data can and cannot support.
Read the worked answer
A test asks 'does it meet the specified behaviour?'; a playtest asks 'what is the experience, and why?'. So no pass/fail oracle: I would form hypotheses first, for example 'the boss's third phase adds a mechanic the player has not been taught', then design sessions that can distinguish that from 'the reward loop is too slow' or 'attempts are too long'.
Design: recruit 12-16 players matching the beta profile, split into the current build and one with a single tuning change (telegraphed phase-three tell), and observe remotely with recording. Measure attempts-to-clear distribution, time per attempt, quit point, retry latency, and a short post-session interview; log telemetry at the same points so lab and field data can be compared.
The data can support 'there is a difficulty spike at this encounter, and this change reduces it'. It cannot support 'this is a bug', and it cannot prove the game is fun. It also cannot generalise from 16 players to 4,000, so I would pair it with an A/B telemetry read after the tuning change ships to a small cohort, then regression-test that the change did not break encounter logic.
3. Set exit criteria for the iOS 15.4 crash and explain why 'zero crashes' is the wrong criterion.
Read the worked answer
'Zero crashes' is unmeasurable and unfalsifiable: you cannot run enough cold starts to prove absence, and it gives no basis for deciding whether the remaining risk is acceptable. It also treats a 0.02% failure on 2% of the install base the same as a 20% failure on 60%.
Better: define crash-free sessions as the primary metric, with a threshold and a sample. For example, at least 2,000 cold starts across at least six iOS 15.4 device models with crash-free sessions >= 99.5%, plus no crash signature in the top 10 by volume on the crash backend, plus a confirmed root cause or a documented workaround. Add a device-share weighting: if iOS 15.4 is 8% of the audience, a 1-in-55 crash there is worth more test effort than a 1-in-500 crash on iOS 17.
I would also make the criteria time-boxed and state the fallback: if the root cause is not found by the code freeze, the decision escalates with the residual risk quantified (expected crashes per 10,000 sessions) rather than being silently accepted.