Balance check outside the transaction

The check happens outside the transaction, on a different snapshot, in a different operation. Between the read and the $inc, another transfer can debit the same account. The transaction then sees the current state and the guard has already passed or failed on stale data - so the guard is enforced against a snapshot that no longer exists. The false negatives come from the same race in reverse: the check reads a balance mid-flight, before a concurrent credit commits.

The fix is to move the decision inside the transaction and let the storage engine arbitrate. Start the transaction with readConcern: "snapshot", read the balance inside the transaction, and write. Because the transaction reads a consistent snapshot and writes with conflict detection, a concurrent write to the same account causes a WriteConflict rather than a lost update - that is exactly the mechanism you want. Alternatively, if the transfer truly is a single-account debit plus a single-account credit and the guard is simple, use one atomic conditional update per side: findOneAndUpdate({_id: src, balance: {$gte: amount}}, {$inc: {balance: -amount}}) and treat a null result as insufficient funds. That removes the read-modify-write window entirely and needs no transaction.

Whichever you choose, do not treat a client-side read as a lock.

These are two different failures and need two different loops.

TransientTransactionError means the transaction was aborted by the server (write conflict, primary stepdown) and nothing was committed. The correct response is to retry the entire transaction body from the start, including re-reading the balances - not just the commit - with a small randomized backoff. Under 400 ops/s on 30 hot accounts, conflicts are expected; exponential backoff with jitter is what keeps a thundering herd from re-colliding every time.

UnknownTransactionCommitResult means the commit was sent but the outcome is unknown. The correct response is to retry commitTransaction() on the same session, not to re-run the body. The driver retries the commit with the majority write concern to establish the outcome.

Safety across retries requires idempotency, because a retry after "unknown" can only be safe if applying the transfer twice is impossible or harmless. The robust answer: give each transfer a client-generated transferId and put a unique index on it in the ledger collection, then insert the ledger row inside the same transaction as the balance updates. A duplicate apply then fails with a duplicate-key error, which the retry loop can recognise as "already done" instead of double-debiting. This is the only thing that explains and prevents the duplicate rows from the earlier timeout.

w: 1 acknowledges after the primary applies the write. If the primary steps down before the change replicates, that transaction is rolled back - and any application that read the acknowledged result has seen money move that then un-moves. For a payments ledger, use {w: "majority", j: true} on the transaction so an acknowledgement means the data survives a failover. The cost is latency, which is why it must be a deliberate decision rather than a default.

Other things to verify:

• Transaction lifetime: the default transactionLifetimeLimitSeconds is 60. If any code path does network I/O or user prompting inside the transaction, it will be aborted mid-flight. Keep transactions short and never make an outbound call inside one.
• Oplog sizing: a transaction's writes must fit in a single oplog entry. Long transactions with many operations can hit the 16 MB oplog entry limit, especially if documents are large.
• Read concern pairing: snapshot reads inside a transaction combined with w: 1 gives a false sense of durability. Match them.
• Unhandled-error paths: the 0.4% crashes are a code problem, not a database problem. Every transaction should live inside a helper with both retry loops, a maximum attempt count, and a clear log line when it gives up, so a persistent conflict storm is visible instead of silent.
• Hot-account contention: 30 accounts taking most of the traffic will keep producing write conflicts no matter how correct the retry loop is. If commit latency on those accounts matters, consider making the hot aggregate (a running balance) a separate document that uses $inc, or queue the ledger writes per account.