Host Migration at Two Percent Packet Loss

The current setup reproduces exactly one network: gigabit, sub-millisecond latency, no loss, full-cone NAT, one office, identical hardware. The reported symptoms all live in the variables you are holding constant.

Environment:

• WAN emulation on the host uplink, since a listen server's upload is the shared bottleneck: latency 40, 120 and 250 ms; jitter plus or minus 30 ms; loss 0.5, 2 and 5%; reordering; and an upload cap of 1-3 Mbps to mimic residential upstream saturation, which is the most likely cause of the 400 ms spikes.
• A NAT matrix run as a real matrix: full-cone, restricted-cone, port-restricted and symmetric, with at least one symmetric peer per lobby and a symmetric-host configuration, because the spikes cluster around same-ISP hosts and symmetric NAT.
• Geographic distribution: at least two cloud regions and a mix of real client ISPs rather than one office link.
• Player behaviour: scripted 20-minute sessions at realistic entity counts, not idle clients.

Method: automate the migration trigger (host quits via menu, host process killed, host uplink dropped, host console suspended) crossed with game state (mid-wave, during a boss, mid-loading, mid-save) and repeat each cell thirty or more times. You cannot measure a 70% success rate with one-off manual tests; you need the denominator.

Define it as a set of invariants with time bounds, all observable or instrumentable:

• All remaining clients converge on the same authoritative world state within ten seconds of the host leaving. Compare a state hash covering entity positions and health, wave index and remaining enemy count.
• Progress is preserved for every client: run time, currency, inventory and objective state match the pre-migration snapshot.
• Exactly one new host is elected, by a documented deterministic rule, and no client is left believing it is host.
• No duplicated entities, no double-awarded loot, no resurrected enemies.
• If the old host reconnects, it joins as a client and does not fork the session.
• A client that cannot migrate gets a defined, recoverable path such as a rejoin prompt that restores the run, not the main menu with progress lost.

Anything not on this list becomes an argument in the field, which is exactly why players report 'I lost my run' while QA reports 'migration passed'. Write the invariant list into the test plan, instrument it in the build, and log a pass rate per trigger-and-state cell into telemetry so the next patch can be judged against a number.

Ask, in this order: session or run ID, timestamp and region, plus the client log bundle. Most of this is answerable from telemetry without the player. Then: was it during loading, mid-mission, or at host migration; did the other players drop too; platform, approximate NAT type, ISP, and whether they were the host; every session or intermittent; and did it start after the last patch.

Triage against the existing data rather than treating the report as a new entity. Cluster by failure point. A loading-timeout signature explains the 15% seven-player starts. Symmetric-NAT hosting explains the ping spikes. Migration failure explains lost runs. A report that matches a known signature joins that bug as another data point, which is useful evidence; a report with a genuinely new signature and no logs goes into a low-confidence bucket, not into closed-as-unreproducible.

Promotion rule, stated in advance: three or more independent reports sharing a signature, or one with a log bundle that pins the failure point, becomes a defect with the telemetry query attached instead of a screenshot.

Close the loop with the player regardless. Whether they bother reporting next time depends on whether the last report vanished into silence.