One heartbeat partition freezes every event-time window
clickstream in Confluent Cloud has 16 partitions and carries roughly 45,000 events/s. A Flink SQL statement aggregates it into 1-minute tumbling windows and writes per-window counts to a dashboard topic. Partition 11 is fed by a legacy service that is being decommissioned slowly: it emits one synthetic heartbeat record every 90 minutes, stamped with the current wall clock. The other 15 partitions are continuously busy. table.exec.source.idle-timeout is still at its default. Job parallelism is 8, checkpoints run every 60 s and finish in about 4 s, there is no backpressure, and Kafka consumer lag is flat. Yet the dashboard freezes for long stretches and then jumps forward: a batch of 1-minute windows lands at once, roughly every 90 minutes. Records are never lost from the source topic, watermarks are enabled on event_time, and no exceptions appear in the logs.
1. Explain why the windows stall even though 15 of 16 partitions are continuously busy. Where exactly is the watermark held back, and why does the stall always end in a burst?
Read the worked answer
A watermark in Flink is a minimum, never an average. Each Kafka source subtask emits a watermark equal to the minimum event-time watermark across the splits assigned to it, and every downstream operator takes the minimum across its input channels. With parallelism 8 over 16 partitions, the split for partition 11 sits on one source subtask. Because that split keeps receiving records - one every 90 minutes - Flink does not consider it idle, so its watermark is pinned at heartbeat_time - 5s. That single value caps the job-wide watermark, and window timers for every minute after it cannot fire. When the next heartbeat arrives, the watermark leaps roughly 90 minutes, all suppressed windows fire at once, and the dashboard catches up. Checkpoints, lag and backpressure all look healthy because the data path is fine; only event-time progress is blocked by one slow split. The 4 s checkpoints prove state size is not the issue.
2. What is the smallest configuration change that unblocks the windows, and what correctness risk do you accept by making it? Name one alternative that avoids the risk.
Read the worked answer
Set table.exec.source.idle-timeout = '60 s' (the DataStream equivalent is WatermarkStrategy.withIdleness). After 60 s without records, that split is marked idle and excluded from the watermark minimum; when the heartbeat arrives it becomes active again. Two consequences must be accepted. First, the setting is per split and job-wide, so a genuinely stalled partition is masked in the same way - pair it with alerting on per-partition numRecordsInPerSecond and on watermarkLag. Second, records from partition 11 that arrive after the watermark has moved past them are late. Flink SQL window aggregations have no allowed-lateness knob, so they are dropped silently. If those legacy records carry business value, either switch to the DataStream API with allowedLateness plus a side output for late events, or isolate the legacy producer onto its own topic so a 90-minute-cadence stream never shares a watermark with the main flow. The config change is one line; the trade-off is a silent late-data policy.
3. An engineer suggests simply retiring the heartbeat producer so that partition 11 stays completely silent. Does that fix the problem? Justify the answer in terms of how Flink advances watermarks.
Read the worked answer
No - with idle detection still disabled it makes the failure permanent. Flink advances a split's watermark only when a record is emitted from it. A split that never emits keeps its watermark at the initial value (effectively negative infinity) and still participates in the per-subtask minimum and therefore in every downstream operator's minimum. So the current 90-minute heartbeat is the only thing that ever releases the watermark; deleting the producer while table.exec.source.idle-timeout remains 0 freezes event time entirely and no window ever fires. The correct sequence is: enable idle detection and confirm recovery using currentInputWatermark and watermarkLag, verify the dashboard emits on schedule, and only then retire the legacy producer and reduce the partition count. Retiring the producer first converts a periodic stall into a total one, and it would also be invisible to lag and backpressure metrics.