Flink Stream Processing and Real-Time Development
Time and state
Flink streaming jobs process unbounded data. Event time uses timestamps in records; processing time uses the system clock. Watermarks advance event time and trigger window evaluation. Windows can be tumbling, sliding, or session. State is keyed and checkpointed; checkpoints provide exactly-once state recovery, while savepoints are manual, portable snapshots. Barriers flow through the topology to align checkpoints. TaskManagers provide slots; slot sharing lets subtasks of a pipeline share a slot.
Worked example
A 5-minute tumbling event-time window has a 2-minute watermark delay and 1-minute allowed lateness. A record with event time 10:00:00 is held until the watermark passes 10:05:00; if it arrives after that but before allowed lateness ends, it still updates the window. After the lateness period, it is dropped unless a side output is configured.
Tuning and traps
- Backpressure often comes from a slow sink or skewed key; check checkpoint alignment time and busy/idle metrics.
- Idle partitions can hold back watermarks; configure idleness where appropriate.
- RocksDB state backend suits large state; HashMapStateBackend suits small state.
- Checkpoint storage must be durable and fast; a timeout is not fixed by only increasing the interval.
- Do not use processing time when results must be reproducible.