Confluent Apache Flink Certificate cheat sheet

Confluent's Flink offering runs SQL and Table API statements against Kafka topics catalogued as tables; CREATE TABLE statements declare a connector, format, and schema, and the topic is the physical storage behind the logical table.

A table can be a source, a sink, or both; INSERT INTO writes to a sink table, and statement sets submitted together execute as one job with shared state.

Watermarks in DDL are declared with WATERMARK FOR column AS expression; a watermark lags event time and drives event-time windows, timers, and idleness handling.

Processing-time operations use PROCTIME() as a computed column; event-time operations require a TIMESTAMP_LTZ or TIMESTAMP column with a declared watermark.

Temporal joins against a versioned or lookup table use FOR SYSTEM_TIME AS OF, enabling enrichment of an event stream with the table state valid at the event's time.

Changelog semantics: a table backed by a Kafka topic can be interpreted as append-only, upsert (with a primary key), or retract stream; the chosen mode determines whether downstream aggregations can consume it.

Formats commonly used include Avro, Protobuf, and JSON with a Confluent Schema Registry URL; schema evolution compatibility is enforced by the registry, not by Flink.

Statements are submitted through the Confluent Cloud Flink SQL workspace or the REST/CLI interfaces; a statement has a lifecycle of pending, running, completed, or failed, and results can be consumed from a result topic.

Exactly-once end-to-end requires a checkpointed source (Kafka consumer with checkpointing enabled) plus a sink with transactional or idempotent semantics; Kafka sinks achieve this via two-phase commit on checkpoint completion.

Checkpoints are periodic asynchronous snapshots of operator state coordinated by the JobManager; checkpoint interval, timeout, and minimum pause between checkpoints are the primary tuning knobs.

State backends: hashmap state lives on the TaskManager heap (or off-heap), while RocksDB stores state on local disk and supports incremental checkpoints, which matters for large keyed state.

Keyed state is partitioned by key group; rescaling redistributes key groups across subtasks, so parallelism changes do not lose state as long as key groups are reassigned.

Event-time windows fire when the watermark passes the window end; allowed lateness permits late elements to still update results, and side outputs can capture elements dropped as too late.

Watermark strategies: periodic (bounded out-of-orderness, monotonously increasing) or custom; idleness prevents an idle partition from stalling the combined watermark of a source.

Savepoints are user-triggered, versioned snapshots used for upgrades, rescaling, and migration; they differ from checkpoints in lifecycle and are not automatically cleaned up.

A job's restart strategy (fixed-delay, exponential-delay, failure-rate) determines recovery behavior after task failure; checkpoint retention and externalized checkpoints preserve state across job cancellation.

The Kafka source connector reads topic partitions and assigns them to subtasks; partition discovery can be enabled so newly created partitions are picked up without a job restart.

Source options include starting offsets (earliest, latest, committed group offsets, or specific timestamps), topic and topic-pattern subscription, and a consumer group id used for offset commit and monitoring.

The Kafka sink writes records with optional key/value formats; exactly-once delivery uses the transactional id prefix and requires the checkpoint interval to bound transaction duration.

Schema Registry integration is configured by URL and subject naming strategy (topic-name or record-name); the registry validates compatibility for the format used.

Avro, Protobuf, and JSON Schema are the schema-bearing formats; a raw format bypasses schema handling and treats bytes as opaque.

Upsert Kafka sink requires a primary key and emits tombstones (null value) for deletions, letting a compacted topic represent the changelog of a table.

Metadata columns such as partition, offset, timestamp, and headers can be exposed as readable columns on a Kafka source table for downstream processing.

Watermarks cannot be produced from a Kafka source without an event-time column in the payload or a metadata timestamp; using the record timestamp requires declaring it explicitly.

Compute pools provide the TaskManager capacity for Flink statements; a statement is assigned a pool, and pool size bounds the maximum parallelism available.

Statements are versioned; updating a statement creates a new version, and the previous version can be stopped with a savepoint to preserve state for the upgrade.

A statement stopped with a savepoint can be restarted from that savepoint, which is the supported path for code changes that are state-compatible.

Metrics exposed include numRecordsIn/Out, currentInputWatermark, checkpoint duration and size, and backpressure indicators; watermark stagnation usually signals an idle or stalled source partition.

Autoscaling adjusts TaskManager resources based on utilization; maximum parallelism of the job is still fixed by the statement's configuration and operator parallelism settings.

UDFs can be written in Java or Python and registered for use in SQL; Python UDFs run in a separate worker environment from the JVM operators.

Resource isolation is per compute pool; multiple statements sharing a pool compete for the same TaskManager slots.

Billing and quotas relate to compute pool capacity (CFU) and consumed Kafka throughput, so idle statements still consume pool resources unless stopped.

Tumbling windows are fixed, non-overlapping intervals declared with TUMBLE(timestamp, interval); hopping windows overlap and are declared with HOP(timestamp, slide, size).

A windowing table function must be used in the GROUP BY clause together with the window_start and window_end columns it produces.

Cumulative windows use CUMULATE(timestamp, step, max_size) and emit progressively growing aggregates within a maximum window size.

Deduplication uses ROW_NUMBER() OVER (PARTITION BY key ORDER BY time) and filters to row number 1; the pattern requires an event-time attribute and produces an upsert stream.

Interval joins require at least one time attribute and an equality predicate plus a time-range condition; regular joins without time bounds retain unbounded state and are discouraged in streaming.

Top-N per key uses ROW_NUMBER() with an ORDER BY and a WHERE rank <= n filter; the output is a retract or upsert stream depending on the sink.

OVER aggregations compute running aggregates over a range or row interval; RANGE BETWEEN INTERVAL requires an ordered time attribute.

MATCH_RECOGNIZE performs complex event processing with PATTERN, DEFINE, and MEASURES clauses, detecting sequences such as A followed by B within a time constraint.

TIMESTAMP and TIMESTAMP_LTZ differ: the former is a wall-clock timestamp without zone semantics, while the latter is an instant in the session time zone and is required for some time functions.

TO_TIMESTAMP converts a string using a format pattern; the resulting type is TIMESTAMP, and epoch conversion uses TO_TIMESTAMP_LTZ.

Built-in functions include string, temporal, conditional, and aggregate functions; window aggregates are ordinary aggregates applied over a windowing table function.

User-defined functions must be registered before use and are scoped to the session or catalog; overloaded signatures are resolved by argument types.

Python UDFs have higher per-call overhead than Java UDFs; vectorized or batch invocation reduces the cost for row-heavy transformations.

Type coercion follows SQL rules, but implicit casts between string and timestamp can silently produce nulls if the pattern does not match.

JSON functions such as JSON_VALUE and JSON_QUERY extract scalars and objects from string columns without declaring a full schema.

Null handling: aggregates ignore nulls, joins do not match null keys, and COALESCE or IFNULL is used to substitute defaults explicitly.