Building Data Pipelines with Kafka
Connect building blocks
A worker runs connectors, each split into tasks (tasks.max) that do the actual copying. In distributed mode the cluster state lives in Kafka itself: config.storage.topic, offset.storage.topic (compacted), and status.storage.topic. Standalone mode keeps offsets locally and cannot scale or fail over.
Converters serialize key/value bytes (Avro, Protobuf, JSON Schema) and are worker-level config, not per-connector - a common gotcha when one connector needs a different format. Single Message Transforms (SMTs) are stateless, per-record, applied in declaration order; they can rename fields, mask data, filter, or route to another topic, but they cannot aggregate, join, or hold state across records.
Error handling: errors.tolerance=all plus errors.deadletterqueue.topic.name and errors.log.enable lets a sink connector skip bad records instead of failing the task. Without it, one poison record stalls the pipeline.
Replication: MirrorMaker 2 replicates topics between clusters, with offset translation, heartbeats, and checkpoints so consumers can resume.
Worked example - source connector offsets
A JDBC source connector tracks its position by a watermark column (timestamp.column.name or incrementing.column.name), stored in the offset topic. If the watermark column is not monotonic or not indexed, records are silently skipped or duplicated. Rule of thumb: the source's offset must be durable and strictly increasing; anything else requires a primary-key-based incrementing strategy.
Trap list
tasks.max higher than the source's ability to split work does nothing; SMTs run after the converter, so they see typed Structs for Avro; Connect does not guarantee exactly-once across arbitrary sinks - design sinks idempotently.