Confluent Kafka Developer flashcards
Confluent Kafka Developer · Build recall with a focused review session.
32 cards. Read the term and the definition here. The deck at the top of this page is optional.
- Partition ordering guarantee
- Kafka guarantees record order only within a single partition. Records in different partitions have no defined relative order, so ordering requires the same key and therefore the same partition.
- Consumer group rebalance
- A rebalance reassigns partitions among consumers in a group when membership changes or subscribed topics change. During the rebalance, partition ownership moves and processing may briefly pause.
- min.insync.replicas
- A topic or broker setting that specifies the minimum number of in-sync replicas that must acknowledge a write when the producer uses acks=all. If fewer replicas are available, the produce fails.
- Log compaction
- A cleanup policy that retains at least the most recent record for each key and removes older values. It is useful for changelog and state-recovery topics where only the latest value matters.
- Offset
- A monotonically increasing identifier assigned to each record within a partition. Consumers track offsets to know which records have been processed and where to resume after restart.
- Idempotent producer
- A producer mode that attaches sequence numbers so brokers can discard duplicate retries. It prevents duplicates caused by network retries but does not provide cross-partition transactional guarantees by itself.
- acks=all
- A producer acknowledgement setting requiring all in-sync replicas to acknowledge a write. Combined with min.insync.replicas, it reduces the chance of data loss during broker failure.
- KRaft metadata mode
- A Kafka metadata architecture where controller nodes manage cluster metadata through a Raft log instead of ZooKeeper. It simplifies deployment and improves metadata scalability and failover.
- Key selection
- The record key determines the target partition through the partitioner. Choosing a stable business key preserves per-entity ordering and enables co-partitioned joins across topics.
- max.poll.interval.ms
- The maximum time allowed between calls to poll. If processing exceeds this interval, the consumer is considered failed, a rebalance occurs, and its partitions are reassigned.
- Manual offset commit
- A consumer pattern where the application commits offsets explicitly after successful processing. It supports at-least-once delivery and avoids marking records complete before side effects finish.
- ErrorHandlingDeserializer
- A wrapper deserializer that catches deserialization failures and passes the record to an error handler. It prevents a single malformed record from stopping the consumer poll loop.
- Transactional producer
- A producer configured with a transactional.id that can atomically write records across partitions and commit consumer offsets. It enables exactly-once processing when consumers read committed data.
- read_committed isolation
- A consumer isolation level that returns only records from committed transactions and filters aborted transactional records. It is required for exactly-once semantics on the read side.
- Co-partitioning
- A requirement for joining two topics in a stream processor: they must have the same partition count and use matching key partitioning so related records land in the same task.
- Idempotent consumer
- A consumer that safely handles duplicate deliveries by tracking processed keys or offsets in durable storage. Repeated records then produce the same final state without duplicate side effects.
- Kafka Connect worker
- A JVM process that runs connectors and tasks, manages offsets and configuration, and distributes work across a cluster. Workers coordinate through Kafka internal topics.
- Source connector
- A Kafka Connect plugin that reads data from an external system and writes records into Kafka topics. It can capture full data or incremental changes.
- Sink connector
- A Kafka Connect plugin that reads records from Kafka topics and writes them to an external system such as a database, object store, or search index.
- Single Message Transform
- A Kafka Connect feature that modifies, filters, routes, or masks individual records as they pass through a connector. It operates per record without aggregations or joins.
- Converter
- A Kafka Connect component that translates between Connect internal data structures and serialized bytes such as Avro, JSON Schema, or Protobuf. It often integrates with Schema Registry.
- Dead letter queue
- A destination topic or store for records that repeatedly fail processing. It allows a pipeline to continue while preserving failed records for inspection and reprocessing.
- Connect internal topics
- Kafka topics used by Connect to store connector configuration, source offsets, and task status. They are named with config.storage.topic, offset.storage.topic, and status.storage.topic settings.
- Change data capture
- A pipeline pattern that streams row-level inserts, updates, and deletes from a database into Kafka. It typically uses one topic per source table and preserves operation metadata.
- Schema Registry subject
- A namespace under which schema versions are registered, commonly derived from a topic name plus key or value. Compatibility rules apply per subject.
- BACKWARD compatibility
- A compatibility mode where consumers using a new schema version can read data written with older versions. It permits adding optional fields and removing fields in controlled ways.
- FULL compatibility
- A compatibility mode requiring both backward and forward compatibility, so new schemas can read old data and old schemas can read new data. It is the strictest common mode.
- KStream versus KTable
- A KStream represents an append-only sequence of independent records, while a KTable represents a changelog-backed table holding the latest value for each key.
- GlobalKTable
- A Kafka Streams table fully replicated to every stream task on every instance. It enables joins with a stream without requiring co-partitioning of the input topics.
- ksqlDB persistent query
- A continuously running query that reads from streams or tables and writes results into a new Kafka topic. It keeps processing until explicitly terminated.
- RBAC
- Role-based access control assigns roles to principals at scopes such as organization, environment, cluster, topic, or consumer group. It governs authorization across Confluent components.
- Service account
- A non-human identity used by applications, connectors, or automation to authenticate. It is assigned roles and API keys instead of using a human user account.