Eighteen Days to Submission
Your studio is eighteen days from submitting a console build to the platform holder's technical requirements review. The title is a single-player action game with an optional online co-op mode. You have two certified development kits and one retail kit, the latter supplied by the publisher and identical to the configuration the holder will test on. Four issues are open. (a) Suspend/resume fails on the retail kit after roughly six hours suspended, returning to the dashboard instead of the game; it passes on both devkits. (b) Signing the active user out during a co-op session leaves the other player frozen until they force-quit. (c) With storage full, the game's save prompt reports success but writes nothing. (d) Disconnecting the controller during a non-skippable cutscene pauses correctly and shows the right reconnect prompt, but the prompt text is truncated at 720p. An automated check that the title must respond to the system's return-to-home request within a fixed time budget fails on two of ten attempts.
1. Rank the five issues in the order you would fix them, and justify the ranking using certification risk and user impact rather than effort.
Read the worked answer
Rank by certification risk multiplied by user impact:
- (c) Storage full but the save reports success. Silent data loss is the worst player outcome and exactly the kind of requirement holders test deliberately. A title that loses saves fails review and damages the brand.
- (b) User sign-out freezing the second player until force-quit. An unrecoverable hang requiring a system-level recovery is a hard requirement failure, and it harms a player who did nothing wrong.
- (a) Suspend/resume returning to the dashboard after six hours on retail hardware. High certification risk because it is deterministic on the exact configuration the holder uses while passing on devkits, which means your internal evidence is misleading you. It is a graceful failure, though, not a hang and not data loss.
- The automated return-to-home timing check failing two in ten. Trivial to a player, but the holder runs it once on the submitted build. A 20% failure rate is a real chance of failing an entire submission on a technicality, so it must be fixed or formally mitigated well before the day.
- (d) Truncated prompt at 720p. Cosmetic, and probably a text-fit change measured in minutes.
Note what is absent from the ranking: effort. (c) may be a one-line fix and (a) may take three days; that changes sequencing, not importance.
2. Describe the suspend/resume test procedure you would run to establish the true failure rate and its boundary, including the states that must be covered.
Read the worked answer
Cover the state dimension before the duration dimension, because the current evidence (600-plus minutes, one state) is a single point.
Suspend points: main menu; mid-level with no activity; mid-cutscene; during a manual save and during an autosave; during a level transition or loading screen; in matchmaking and in an active co-op session; with the system UI or an OS overlay open; with the controller disconnected.
Durations: 5 minutes, 30 minutes, 1 hour, 6 hours, 24 hours. Since 6 hours is the reported failure, bracket it at 4, 6 and 8 hours to find the boundary rather than confirming a single point.
External changes during suspension: a different user signs out and in; storage fills; the network drops; the disc is removed on the retail kit; the system reboots or the game updates.
Each cell: ten repetitions on the retail kit, retail firmware, the same quality preset the holder will use, captured on video with system-level logs alongside game logs and a memory high-water trace. The pass condition is that the game returns to the same gameplay state within a stated time budget, the save is intact, and any online session survives or recovers cleanly. Report a rate per cell. 'Fails sometimes' is not a defect report the developer can act on.
3. The automated timing check fails two in ten. The lead developer says it is flaky and will pass on the day. How do you respond and what do you do next?
Read the worked answer
Flaky is a statement about our knowledge, not about the check. The holder runs it once on the build we submit, so two in ten is roughly a one-in-five chance of failing the submission and losing the slot. For a check that probably takes a day to stabilize, that is not a bet worth taking.
Response:
- Reproduce it in our own harness and produce a distribution rather than a pass/fail. If the 99th percentile is above the budget, we have a real performance problem. If only two runs cross, find out what those two were doing: cold start, background content install, shader compilation, a network call on the request path.
- Reduce the variance: precompile shaders, remove synchronous disk reads from the home-request path, push non-essential work off it.
- If it cannot be made deterministic in time, raise it as a known issue with the distribution as evidence and ask the publisher's account contact how intermittent failures are treated. Do that this week, not on submission day.
- Never plan on it passing. That is a coin flip with a milestone attached, and if it lands badly the post-mortem will ask why the data was available and unused in week three.