Java transactions: two error labels, a duplicated outbox event, and a 60-second batch

The two labels mean different things, and treating them the same is the bug.

• TransientTransactionError: the transaction aborted (write conflict, primary stepdown) before or during its work. Recovery is to startTransaction() again on the same session and re-run every operation, with backoff. startTransaction() implicitly aborts the old transaction number and allocates a new one.
• UnknownTransactionCommitResult: the commit was sent but the reply was lost. It may have succeeded. Re-running the body would apply the transfer twice. Recovery is to re-send commitTransaction() on the same session until you get a definite result or hit a retry limit; the server deduplicates by transaction number, so a repeated commit is safe.

int txnRetries = 3, commitRetries = 3;
while (true) {
  session.startTransaction(TransactionOptions.builder()
      .readConcern(ReadConcern.SNAPSHOT)
      .writeConcern(WriteConcern.MAJORITY).build());
  try {
    debit(session, from, amount);
    credit(session, to, amount);
    writeOutbox(session, transferId);
    while (true) {
      try { session.commitTransaction(); break; }
      catch (MongoException e) {
        if (e.hasErrorLabel("UnknownTransactionCommitResult") && commitRetries-- > 0)
          continue;              // re-commit only
        throw e;
      }
    }
    break;
  } catch (MongoException e) {
    if (session.hasActiveTransaction()) session.abortTransaction();
    if (e.hasErrorLabel("TransientTransactionError") && txnRetries-- > 0)
      continue;                  // whole transaction again
    throw e;
  }
}

Two supporting points: bound the commit with maxCommitTimeMS so a stuck commit cannot pin the request thread, and make the HTTP layer replay the same requestId on retry - a fresh requestId defeats the outbox deduplication below.

Root cause chain. The transaction commits, the reply is lost, the label-driven path or the gateway retries the whole request with a new requestId, and the outbox insert creates a second document because nothing identifies it as the same event.

Fix inside the transaction: give the outbox document a deterministic _id derived from the business key, for example transferId + ":completed" or a hash of (requestId, eventType). The insert then either succeeds once or fails with E11000.

There are two acceptable behaviours on the duplicate:

• Abort on duplicate. The E11000 aborts the transaction, which also rolls back the debit and credit - correct if the only reason you are here is a retry of an already-applied transfer. The caller sees an idempotent success rather than a second transfer.
• Skip the side effects. Inside the same snapshot, look up the outbox _id first and return early without debiting or crediting. Cleaner for the caller, one extra point read on _id.

Why it matters beyond the duplicate row. The poller does something like find({published:false}) and marks documents published; a duplicated event is delivered to the downstream service twice. Consumers must be idempotent too, and the deterministic outbox _id is what gives them a key to deduplicate on (e.g. a unique index on eventId on the consumer side).

Do not weaken the write concern to work around this. A w:1 commit can be rolled back on stepdown, leaving the poller with an event whose debit no longer exists - a strictly worse inconsistency than a duplicate. Keep MAJORITY and make the event idempotent.

What the limit is. transactionLifetimeLimitSeconds defaults to 60 and is enforced server-side; it is not a client timeout. A single transaction running 40,000 updates will be aborted once it crosses that wall-clock age.

Why the batch is a design smell, not just a slow query:

1. Snapshot pinning. The transaction holds a WiredTiger snapshot and pins the history it needs; a long-running multi-document transaction forces the cache to retain old versions, which causes eviction pressure and slows reads for unrelated traffic.
2. Lock contention. 40,000 document-level writes held open for a minute will collide with the nightly load and produce write conflicts (TransientTransactionError) that abort the entire body.
3. All-or-nothing blast radius. Any failure discards all 40,000 updates, and retrying means starting from zero.

Raising the limit moves the failure further out and worsens the cache retention, the lock window, and the retry cost. It does not make the transaction viable.

Correct approach - chunked, resumable, idempotent:

• Batch by 500–1,000 documents, commit each batch, and persist the last processed _id (in a checkpoint document) so a re-run resumes instead of restarting.
• Make each update conditional on _id (or on a version field) so replaying a batch is a no-op rather than a double-apply.
• Keep the batches outside transactions entirely where possible: a single-document update is atomic on its own, and bulkWrite with ordered:false gives you throughput without a transaction.

Use a multi-document transaction only where atomicity across documents is genuinely required - here it clearly is for the transfer (debit + credit + outbox) and clearly is not for the nightly recompute. Remember also that a transaction needs a replica set (available here) and costs roughly 2–4× the latency of the same writes outside one.