Blue/green Flink jobs, transactional id fencing and doubled revenue
orders-enrich is a self-managed Flink 1.19 job on Kubernetes. It reads orders_raw from Confluent Cloud and writes orders_enriched through KafkaSink with DeliveryGuarantee.EXACTLY_ONCE, transactional.id.prefix = "orders-sink", 30 s checkpoints to S3, and all downstream consumers using read_committed. For a blue/green validation the team deploys v2 with an identical configuration except consumer group orders-enrich-v2 and scan.startup.mode = latest, leaving v1 running. About 90 s after v2 reaches RUNNING, v1 starts dying with ProducerFencedException, restarts, and dies again in a loop; v2 is unaffected. The operator then changes v2's prefix to orders-sink-v2. Both jobs now run cleanly for 40 minutes, after which v1 is stopped - but the revenue dashboard reports roughly double for those 40 minutes and returns to normal afterwards. No records are missing from orders_raw, and no other configuration changed.
1. Explain why v1 was fenced. Why does the failure appear in the healthy job rather than in the newly deployed one?
Read the worked answer
KafkaSink derives each producer's transactional id from the configured prefix plus the subtask index and the checkpoint id, so two jobs configured with the same prefix will eventually construct identical transactional ids. The first time v2 commits a transaction under an id that v1 already used, the broker bumps the producer epoch for that id; the older producer (v1) is fenced and its in-flight transaction is aborted. v1 then fails with ProducerFencedException, restarts from its checkpoint, produces again, and is fenced again, which is the loop you see. It is not a resource, ACL or network problem, so v2 is unaffected - v2 holds the current epoch. The important judgement is that the transactional id is a global name in the Kafka cluster, not a per-job one: any two Flink jobs writing with EXACTLY_ONCE must have distinct prefixes, including during migration windows where both run.
2. After the prefix was changed to `orders-sink-v2` the pipeline stopped failing, but revenue doubled. Where exactly does exactly-once apply here, and which cutover strategy would you have used instead?
Read the worked answer
The EXACTLY_ONCE guarantee is scoped to a single job's sink instances and to the transactions they coordinate with that job's checkpoints. Once v2 has its own prefix it owns a disjoint set of transactional ids, so Kafka sees two independent, individually correct exactly-once producers: each record is written once by v1 and once by v2. Nothing in Kafka, and nothing in read_committed, deduplicates across jobs - read_committed only hides uncommitted and aborted data. Hence 2x. Better cutover options, in order of preference: (1) have v2 write to a shadow topic, compare outputs, then switch consumers atomically and retire v1; (2) start v2 from a savepoint taken from v1 so state and offsets are continuous, instead of a fresh group with latest, which also avoids skipping the history v2 never read; (3) if both must write the same topic, make it compacted and keyed by order_id so downstream reads collapse to last-value-per-key. Choose (1) when the sink is a finance number; choose (3) only when the key is genuinely idempotent.
3. During the fencing loop, were any records lost or duplicated in `orders_enriched`? Reason about the aborted transactions and the checkpointed offsets.
Read the worked answer
For v1's own stream, no records were lost. Transactions are committed only after the corresponding checkpoint completes, so everything v1 had committed is durable, and everything in the aborted in-flight transaction was never visible to read_committed consumers. On restart v1 rewinds to the source offsets stored in its last completed checkpoint, re-processes the records that were in the aborted transaction, and writes them again under a new transaction - exactly-once is preserved across the failure because the aborted output and the rewound input cancel out. What did happen is a processing pause: while v1 looped, no output was produced on its stream, so consumer lag on orders_enriched spiked. The genuine duplication came later, from two live jobs with distinct prefixes, which is a deployment-level mistake, not a consequence of the fencing. The subtlety worth remembering: fencing causes availability loss, not data loss; two healthy jobs cause duplication.