Confluent Apache Flink Certificate flashcards

32 cards. Read the term and the definition here. The deck at the top of this page is optional.

Watermark definition

A watermark is a monotonically advancing event-time marker asserting no events earlier than it should arrive. It triggers event-time windows, timers, and cleanup of state for expired windows.

WATERMARK FOR clause

A DDL clause that declares a watermark on a time column, specifying the delay expression. It converts a plain timestamp column into an event-time attribute usable by windows and temporal joins.

PROCTIME computed column

A computed column defined as PROCTIME() that yields the processing-time timestamp of the operator. It enables processing-time windows and lookups but is not deterministic on replay.

Temporal table join

A join using FOR SYSTEM_TIME AS OF that enriches each event with the version of a lookup table valid at that event's time. It is the standard pattern for dimension enrichment in Flink SQL.

Changelog stream modes

A dynamic table can be emitted as append-only, upsert with a key, or retract with add and retract flags. The mode determines which downstream operators can consume the stream correctly.

Schema Registry subject strategy

The rule mapping a table to a registry subject, typically topic name or record name. It determines which schema is fetched and which compatibility policy applies during evolution.

Exactly-once Kafka sink

The sink commits Kafka transactions only when a checkpoint completes, giving exactly-once delivery. Transaction timeout must exceed checkpoint interval plus recovery time to avoid aborted data.

Checkpoint barrier alignment

Barriers injected at sources flow through operators, which snapshot state upon receiving them. Alignment waits for all input channels, so slow channels increase checkpoint duration and size.

RocksDB state backend

A state backend storing working state on local disk with incremental checkpoint support. It suits large keyed state but adds serialization cost and depends on local disk availability.

Key groups and rescaling

Keyed state is partitioned into key groups, the unit of redistribution. Changing parallelism reassigns key groups to subtasks, preserving state as long as the maximum parallelism is not exceeded.

Savepoint versus checkpoint

A savepoint is a user-triggered, retained snapshot for upgrades and migration; a checkpoint is automatic, periodic, and typically discarded after the next one succeeds. Both capture operator state.

Allowed lateness

A window property permitting elements arriving after the watermark passes the window end to still update results. Elements later than the allowed lateness are dropped or sent to a side output.

Idle source handling

A watermark strategy option marking a source partition idle so it stops holding back the combined watermark. It prevents one silent partition from stalling all event-time progress.

Restart strategy

A policy defining how many times and how quickly a job restarts after failure, such as fixed-delay, exponential-delay, or failure-rate. It bounds recovery attempts before the job fails terminally.

Kafka source offset start

The position from which a Kafka source begins reading: earliest, latest, committed group offsets, or a timestamp. Choosing committed offsets resumes a job without reprocessing or data loss.

Partition discovery

A source option enabling detection of newly added Kafka partitions at runtime. Without it, records written to new partitions are invisible until the statement is restarted.

Upsert Kafka sink

A sink requiring a primary key that writes updates as keyed records and deletions as tombstones. It lets a compacted topic represent the changelog of a dynamic table.

Metadata columns

Readable virtual columns exposing source record attributes such as partition, offset, timestamp, and headers. They allow filtering or routing logic based on Kafka record metadata.

Raw format

A format treating the Kafka value as opaque bytes without schema interpretation. It is used when payload parsing happens inside a user-defined function instead of the connector.

Compute pool

The Confluent Cloud resource unit supplying TaskManager capacity to Flink statements. Its size bounds the parallelism available and is billed independently of statement activity.

Statement versioning

Each update to a SQL statement creates a new immutable version. Upgrading statefully means stopping the current version with a savepoint and starting the new one from it.

Watermark stagnation metric

A monitoring signal that the current input watermark has stopped advancing. Typical causes are an idle partition, a stalled upstream producer, or a watermark delay set too large.

Backpressure indicator

A metric showing an operator cannot keep pace with its input, causing upstream slowdown. Persistent backpressure points to insufficient parallelism or an expensive operator such as a join.

Python UDF overhead

Python user-defined functions cross the JVM boundary per call, adding serialization cost. Vectorized invocation or rewriting hot logic in Java reduces per-record overhead substantially.

Tumbling window

A fixed-size, non-overlapping window declared with TUMBLE over a time attribute. Each event belongs to exactly one window, and results emit when the watermark passes the window end.

Hopping window

An overlapping window declared with HOP specifying slide and size. An event belongs to size divided by slide windows, so smaller slides increase computation and output volume.

Cumulative window

A window declared with CUMULATE that emits progressively growing aggregates at each step up to a maximum size. It suits running totals that reset periodically, such as daily cumulative counts.

Deduplication with ROW_NUMBER

A pattern assigning row numbers partitioned by key and ordered by event time, then keeping row one. It requires a time attribute and produces an upsert stream with one row per key.

Interval join

A join between two streams with an equality predicate and a bounded time-range condition. The time bound lets Flink expire state, unlike an unbounded regular join that retains it indefinitely.

MATCH_RECOGNIZE

A SQL clause performing complex event processing using PATTERN, DEFINE, and MEASURES. It detects ordered sequences across rows, such as a login followed by a failure within a time limit.

TIMESTAMP versus TIMESTAMP_LTZ

TIMESTAMP is a wall-clock value without zone semantics; TIMESTAMP_LTZ represents an instant in the session time zone. Epoch-based conversions and some time functions require the latter type.

Null handling in joins

Equality joins never match null keys, and aggregates skip nulls. Use COALESCE or IFNULL to substitute defaults before joining or aggregating when nulls carry meaning.