Flink Fundamentals and Architecture
This domain tests whether you can reason about how a Flink program becomes a distributed, stateful job: the graph Flink builds, how parallelism and task slots map onto that graph, and how event time drives results. Questions here are usually diagnostic - a job that runs but emits nothing, a job that needs more slots than expected, or a watermark that never advances.
From API calls to a physical job
DataStream and Table API calls only build a logical graph. Nothing is submitted until execute() or executeAsync() is called. Flink translates the graph into a JobGraph and then an ExecutionGraph of parallel tasks. Operators with the same parallelism connected by a forward, non-shuffling exchange are fused into a single task by operator chaining; startNewChain(), disableChaining() and slotSharingGroup() reshape that fusion.
Parallelism and slots
Parallelism is set per operator; keyBy, rebalance and broadcast create network shuffles that let upstream and downstream parallelism differ. Each TaskManager registers a number of task slots, and by default all operators of a job share one slot sharing group, so the slots a job needs equal the highest operator parallelism in the job - not the sum across operators. setMaxParallelism fixes the number of key groups and therefore the ceiling for rescaling from a savepoint.
Event time and watermarks
A watermark tells operators that no events older than T are expected. With bounded out-of-orderness the watermark is the maximum timestamp seen minus the configured delay. Each source subtask (normally one per Kafka partition) generates its own watermark, and an operator with several inputs uses the minimum across them. One idle partition stalls the whole job unless withIdleness(...) is attached.
Worked example
A pipeline reads with parallelism 4, maps, keys by user and windows, then writes with parallelism 2. Chaining plus slot sharing means the job needs 4 slots: four source chains and two sink subtasks all fit inside them. Forcing startNewChain() on the map does not change the total - the maximum is still 4. Suppose the source uses forBoundedOutOfOrderness(Duration.ofSeconds(5)) and events with timestamps 10, 12, 9, 20 arrive. After 20 is processed the subtask watermark is 15, so a tumbling window whose end is 15 can fire.
Traps that cost marks
Assuming chaining changes parallelism; adding operator parallelism to estimate slot needs; treating the watermark as a global maximum timestamp; expecting a window to emit when its last element arrives rather than when the watermark passes the window end; forgetting that keyed windows and keyed state require a preceding keyBy.
Flink SQL and Table API for Streaming
This domain tests continuous SQL over Kafka: table definitions, connector and format options, event-time windowing with table-valued functions, join semantics, and changelog handling. Expect questions that hand you a query and ask why it produces no rows, or why the sink rejects the resulting changelog.
Dynamic tables and continuous queries
A SELECT over a Kafka-backed table never terminates; the result is a dynamic table that updates as rows arrive, and INSERT INTO sink SELECT ... starts an unbounded job. Several inserts can be submitted in one submission with EXECUTE STATEMENT SET BEGIN ... END;, which shares a single job graph.
DDL essentials
Tables are declared with CREATE TABLE ... WITH (...). Typical Kafka options are 'connector'='kafka', 'topic'=..., 'properties.bootstrap.servers'=..., 'properties.security.protocol'='SASL_SSL', 'scan.startup.mode'='earliest-offset', plus independent 'key.format'/'value.format' and their Schema Registry URL options. PRIMARY KEY NOT ENFORCED declares upsert semantics. Event-time processing requires WATERMARK FOR ts AS ts - INTERVAL '5' SECOND; without it, event-time window functions fail.
Windowing with table-valued functions
TABLE(TUMBLE(TABLE orders, DESCRIPTOR(ts), INTERVAL '5' MINUTES)) produces window_start and window_end, grouped with GROUP BY window_start, window_end. HOP is sliding, CUMULATE is growing, SESSION groups by inactivity gap. Rows are emitted when the watermark passes the window boundary.
Joins
A regular join keeps both inputs in state indefinitely and is unbounded. An interval join constrains the time relationship (WHERE o.ts BETWEEN p.ts - INTERVAL '1' MINUTE AND p.ts + INTERVAL '1' MINUTE) and lets Flink expire state. A temporal join against a versioned table uses FOR SYSTEM_TIME AS OF o.proc_time AS p.
Worked example
You aggregate orders per customer in one-minute tumbling windows. Because each window closes exactly once, the output is an append stream and a plain 'connector'='kafka' sink with 'value.format'='avro-confluent' is correct. Drop the windowing and group only by customer: the result is now a changelog with updates and deletes, so the sink must be 'connector'='upsert-kafka' with a declared key and primary key. Sending that changelog to a plain append sink fails or silently duplicates, depending on the format.
Traps that cost marks
Omitting the WATERMARK clause; using the legacy GROUP BY TUMBLE(ts, ...) form; expecting a bare SELECT to run a job; using an append Kafka sink for an updating query; forgetting the Schema Registry URL that avro-confluent requires; assuming a regular join is bounded by watermarks.
DataStream API and Event-Time Processing
This domain tests your ability to build event-time pipelines in Java: assigning watermarks, using the window API, and controlling behavior per key with process functions, timers and managed state. Questions tend to be about timing - when a window fires, when a timer fires, and where a late record goes.
Sources and watermark strategies
Use env.fromSource(kafkaSource, WatermarkStrategy, name). The strategy combines a generator (forBoundedOutOfOrderness(Duration), forMonotonousTimestamps()) with withTimestampAssigner(...) and optionally withIdleness(Duration). Source parallelism should follow partition count; more source subtasks than partitions leaves subtasks idle.
Window API
Keyed windows are built as stream.keyBy(...).window(TumblingEventTimeWindows.of(Time.minutes(5))), then aggregate, reduce or process. Add .allowedLateness(Time) to keep window state after firing, and .sideOutputLateData(tag) to capture records that arrive after that grace period instead of dropping them silently.
Process functions and timers
KeyedProcessFunction gives per-key access to ValueState, ListState, MapState and to timers. ctx.timerService().registerEventTimeTimer(ts) fires when the watermark reaches ts; registerProcessingTimeTimer fires on wall-clock time. Timers are deduplicated per key and timestamp, so store and delete them deliberately.
Worked example
A 10-second tumbling window over [0, 10) has maximum timestamp 9. When the watermark reaches 11, the window fires and emits its result. With allowedLateness(Time.seconds(5)), the window state is retained until the watermark reaches 9 + 5 = 14. An event with timestamp 7 arriving while the watermark is 13 is still merged into the window and produces an updated result; the same event arriving at watermark 15 is routed to the late side output. Without allowed lateness, that record is discarded - a common cause of "missing" data that is easy to overlook.
Traps that cost marks
Registering an event-time timer inside a non-keyed stream; confusing processing-time timers with event-time timers; keeping window state alive forever with generous allowed lateness; forgetting that managed state is only accessible after keyBy; assuming apply runs per element rather than per window; ignoring withIdleness when one partition has no traffic.
State Management, Checkpointing and Exactly-Once Semantics
This domain tests fault tolerance: which state backend to choose, how checkpoints are produced and restored, and what is actually required for end-to-end exactly-once. Expect scenario questions - large state, slow checkpoints, duplicate records after a restart - where you must choose the right setting or explain the observed behavior.
State backends
HashMapStateBackend keeps state on the JVM heap and is fast but bounded by memory and GC. EmbeddedRocksDBStateBackend keeps state off-heap on local disk and supports incremental checkpoints, which is the usual choice for very large keyed state. Set it via state.backend.type; checkpoint data itself goes to durable storage such as a filesystem or object store.
Checkpoints, savepoints, recovery
Sources inject barriers that flow through the graph; aligned checkpointing waits for every input to reach the barrier, while unaligned checkpointing snapshots in-flight data so backpressure does not stall the checkpoint. Key settings are execution.checkpointing.interval, min-pause, timeout, max-concurrent-checkpoints and tolerable-failed-checkpoints. On failure the job restores from the latest completed checkpoint. Savepoints are triggered manually, are portable across code changes and are the mechanism for planned upgrades; checkpoints are internal and not intended for that.
End-to-end exactly-once
Flink gives exactly-once state updates when state is checkpointed and the source can replay offsets. End-to-end exactly-once additionally needs a transactional or idempotent sink. The Kafka sink with 'sink.delivery-guarantee'='exactly-once' writes in transactions that commit when a checkpoint completes, using 'sink.transactional-id-prefix'; changing that prefix across restarts breaks the guarantee.
Worked example
A job holds roughly 800 GB of keyed state with a 60-second checkpoint interval. On-heap state is not viable, so use RocksDB with incremental checkpoints: only new SST files are uploaded, keeping intervals short. If checkpoint duration approaches the interval, increase min-pause rather than shrinking the timeout, and enable unaligned checkpoints when backpressure is the cause. Add state TTL for keys that never expire.
Traps that cost marks
Assuming checkpointing alone gives end-to-end exactly-once; leaving the Kafka sink at its default at-least-once delivery; changing the transactional ID prefix between runs; setting a checkpoint timeout smaller than real checkpoint duration; treating savepoints and checkpoints as interchangeable.
Kafka Connectors, Formats and Schema Registry Integration
This domain tests how Flink SQL and DataStream programs talk to Kafka and Schema Registry: startup modes, serialization formats, changelog connectors, and delivery guarantees. Most questions are configuration forensics - a table reads nothing, a sink duplicates, or a key is missing so records land in one partition.
Source behavior
scan.startup.mode controls where reading begins: earliest-offset, latest-offset, group-offsets, specific-offsets and timestamp. The default is group-offsets, which depends on committed offsets for the configured consumer group, so set the mode explicitly when reproducibility matters. Partition discovery is controlled by scan.partition.discovery.interval.millis; new partitions are picked up at runtime.
Formats and keys
Key and value formats are configured independently: 'key.format'/'value.format'. Available formats include avro-confluent, json, debezium-json, canal-json, maxwell-json, csv and raw. avro-confluent resolves schemas through Schema Registry and needs the registry URL; plain avro does not. Declare 'key.fields' when the key must be extracted from columns, otherwise the key bytes are null and Kafka's default partitioner scatters records.
Changelog connectors and delivery
The kafka connector is append-oriented; an updating query needs 'connector'='upsert-kafka' with a primary key, which writes a compacted topic and can be read back as a changelog. On the sink side, sink.delivery-guarantee accepts none, at-least-once (default) and exactly-once; sink.partitioner chooses between default, fixed and round-robin routing.
Worked example
Debezium Avro change events on customers must be materialized into a compacted topic keyed by id. Read with 'value.format'='debezium-json' (or avro-confluent if the registry holds the schemas) and 'value.debezium-json.schema-include'='false', define PRIMARY KEY (id) NOT ENFORCED, then write with the upsert-kafka connector and 'key.format'='avro-confluent'. Using a plain kafka sink here appends every insert, update and delete as separate records and loses the current-row semantics entirely.
Traps that cost marks
Choosing avro when the registry is the source of truth; omitting the Schema Registry URL; surprising results from the default group-offsets startup mode; sending an updating query to an append sink; forgetting key fields so all records hit one partition; assuming JSON payloads carry schema evolution guarantees.
Deployment, Operations and Monitoring on Confluent Cloud
This domain tests running Flink as a managed service: submitting SQL statements through the workspace, CLI, REST API or Terraform; sizing compute pools measured in Confluent Flink Units (CFUs); and diagnosing a statement that is running but falling behind. Questions are operational - what to change first when lag grows, and what the platform does not let you configure.
Compute pools and statements
A compute pool is provisioned in a cloud region and sized in CFUs, optionally with autoscaling and a maximum CFU ceiling. Statements are submitted into the workspace and reference a compute pool and a database; the platform schedules the job onto that pool, so you do not manage TaskManagers, slots or a flink-conf.yaml. Statement lifecycles move through pending, running, then completed, failed or stopped. The confluent flink CLI (compute-pool create, statement create, statement list, statement describe, statement delete) plus the REST API and Terraform cover the same operations programmatically.
Monitoring what matters
Watch statement status, records in and out per second, watermark progress and source lag. Watermark lag growing while consumption keeps up signals a stalled or idle partition upstream. A statement that fails repeatedly usually carries the cause in its exception log rather than in the metrics, so read the error text before resizing anything.
Scaling levers
Capacity comes from the compute pool, but throughput is usually bounded by partitioning and query shape. A source topic with six partitions cannot feed more than six productive source subtasks. Regular joins and unbounded state inflate checkpoint size and cost; interval joins, state TTL and pre-aggregation reduce it. Adding CFUs helps CPU- or memory-bound statements, not skewed ones.
Worked example
A statement reads a six-partition topic, joins to a lookup table and writes to an output topic. Consumer lag climbs steadily while CFU utilization sits near its ceiling. The first check is partition count: with six partitions, raising the pool ceiling helps only until CPU per subtask is comfortable. Next inspect the join: replacing a regular join with an interval join and a temporal lookup for enrichment shrinks state, shortens checkpoints and often removes the need for more capacity. Raising CFUs first, without fixing the join, usually just moves the bottleneck to checkpoint duration.
Traps that cost marks
Assuming you can set per-operator parallelism or slots directly; treating CFUs as free in cost estimates; expecting a failed statement to restart indefinitely without intervention; assuming self-managed savepoint workflows exist for moving state between environments; resizing the pool when the real limit is source partitions or key skew.