Three Network Symptoms, One Friday Build

Split before fixing The producer is asking for one fix for four problems. Give each its own instrumentation and owner.

• Desync, 1.8%, clustered in one map region. Region clustering points at the map, not the netcode: a collider or navmesh differing between client and server cook, or a locally simulated physics object. Instrument per-actor state-hash comparisons at fixed intervals, logged with actor ID and world coordinates, so the first divergent actor is identified rather than the downstream symptom. Diff the cooked map assets against the server build - a stale cook is the classic cause of exactly this signature.
• Hit registration, RTT > 90 ms, jitter > 25 ms. This is lag compensation, not desync. Log, per shot: client-reported fire time, server rewind window, and the verdict. Then chart hit rate by RTT and jitter bucket. If it collapses only above a jitter threshold, the interpolation delay or rewind buffer is tuned for low jitter.
• Server P99 9 to 45 ms with 400 projectiles. A server performance defect with a clear trigger. Profile the ability at 1, 50, 200 and 400 projectiles; count replicated actors per tick; break down the tick budget. At 45 ms the server is nearly 3× the 16.6 ms tick, which can itself produce hit-registration failures.
• NAT, 19% symmetric. Matchmaking and relay coverage, measured separately from gameplay; it needs relay fallback, not a netcode change.

Ordering insight: fix the tick overrun first, then re-measure hit registration. Otherwise you tune lag compensation against a broken server.

Matrix dimensions RTT at 0, 40, 80, 120, 200 and 300 ms; jitter at 0, 10, 30 and 80 ms; packet loss at 0, 1, 3 and 5%; loss pattern random versus burst; plus a bandwidth cap for the projectile-heavy ability. Full cross is far too large. Use a covering array, and add explicit boundary rows at the thresholds telemetry already flags (90 ms RTT, 25 ms jitter) because that is where real players sit.

Making it reproducible

• Emulate at the network layer with netem or a hardware shaper. Never use a client-side artificial delay, because that hides the server's view of the player.
• Use recorded, scripted matches rather than human play so runs are comparable across builds; log every input with a timestamp.
• Fix the seed and the build, run each condition at least three times, and report variance rather than a single run.
• Attach a capture to every failure; a description alone cannot be re-run.

Pass criteria, written before testing

• Shots inside the rewind window register at or above X% (the no-latency baseline minus a defined tolerance) in every RTT bucket.
• Server tick P99 at or under 16.6 ms in the 400-projectile scenario, with no overrun cascade.
• Zero state-hash mismatches in a 20-minute scripted match at 3% loss.
• No rubber-banding beyond a defined positional error, no permanent stall, correct reconnect after a 5-second outage.
• A mid-match region failover leaves the player in a defined state within a stated time.

Report as condition × metric, so anyone can see which single condition is out of spec.

Community statement Do not blame the cloud migration yet. The evidence is correlational - regions migrated last week, and desync is clustered in one map. Map clustering fits a content or cook problem better than a provider problem. If the statement is wrong, the studio loses credibility twice, once for the outage and once for the excuse, and the community will find the contradiction. Advise instead: name the symptoms specifically, state what is being investigated, give a date for the next update, and commit to a fix only when a fix exists. Silence on a tripled report rate reads as indifference.

Friday ranking

1. Server P99 spike. Known trigger, reproducible in a profiler, plausibly causes the hit-registration reports, and degrades whole matches rather than individuals. Best impact per engineering hour.
2. Hit registration. Second, but re-measure it after the perf fix. Tuning rewind while the server systematically overruns its tick would make it worse, and would also confound the data.
3. Desync on the single map. Contained and diagnosable; if root cause will not land by Friday, removing the map from rotation is a legitimate mitigation provided it is communicated.
4. NAT failures. Real and worth fixing, but it is a relay/fallback feature, not a Friday change. Publish a workaround and a timeline.

Ship what is verified, label what is mitigated, and refuse to let a date force a fix without a repro - an unverified change to netcode on the last day is how you convert three known bugs into an unknown number of new ones.