Real-time dashboard shows stale counts during a burst

The symptom (source healthy, sink fast, but no fresh results) localizes the problem to the Flink job's internal progress, not the endpoints.

• Checkpoint bloat: checkpoint time went 8s → 90s. Long checkpoints during a burst usually mean state is large or backpressure is delaying barrier alignment. If checkpoints fail or time out, the job restarts; on restart it replays from the last successful checkpoint, so downstream sees a frozen or rewound view.
• Repeated restarts confirm the job is not making steady forward progress. Each restart resets window state and re-reads from source offsets.
• Watermark stall: with event-time windows and 30s allowed lateness, if a few partitions lag (storm causing out-of-order or delayed pings), the global watermark is held back by the slowest subtask. Windows never fire, so the serving table gets nothing new even though data is flowing.
• Backpressure from the burst propagates upstream and inflates checkpoint alignment time.

The sink being fast rules out the serving layer; the answer is job-internal state/watermark/checkpoint behavior.

Target both state size and watermark stalls:

1. Reduce state: the 1-minute per-vehicle window is small, but if the job keeps long-lived per-vehicle state or unbounded allowed lateness, state grows. Keep allowed lateness tight (already 30s) and ensure state TTL is set so idle vehicles don't accumulate.
2. Isolate slow partitions: if watermark is held by a lagging source partition, use per-partition/idle-source watermark handling so one lagging vehicle or shard cannot stall all windows. Consider keying by vehicle with sufficient parallelism to spread hot vehicles.
3. Tune checkpoints: increase checkpoint interval slightly and enable incremental/rocksdb checkpoints if state is large, and increase checkpoint timeout so a 90s checkpoint during a burst doesn't fail and trigger restart loops. Monitor checkpoint alignment time as the leading indicator.
4. Backpressure: raise parallelism for the windowing operator or add buffering so a 3x burst doesn't saturate a subtask.
5. Graceful degradation: for the live map, publish last-known position with a freshness timestamp so operators see staleness explicitly rather than a silently frozen map.

Re-test with a simulated 3x burst and confirm checkpoint duration stays bounded and watermarks advance.

They look similar downstream but differ in observable signals:

• Watermark stall only: job uptime is continuous (no restarts), checkpoint duration may be normal, but the current watermark lags event time and window output count is flat. Source consumption continues. Look at watermark lag per subtask: one or a few subtasks lag while others are current → stall.
• Checkpoint/restart cause: job restart count increments, checkpoint duration/failures spike, and output freezes in bursts aligned with restart times. After restart, output may briefly rewind (duplicate or older positions). Check checkpoint history: failed or timed-out checkpoints preceding each freeze.
• Combined: a burst can cause backpressure → slow checkpoints → restart → replay → watermark rewind. Correlate freeze timestamps with restart timestamps and checkpoint durations; if freezes precede restarts and watermark lag grows before the restart, treat backpressure as root cause and both symptoms as consequences.

Instrument: restart count, checkpoint duration/failure, per-subtask watermark lag, and output record rate. The pattern of which moves first identifies the root cause.