Ledger double-spend: replication lag and a retry that mints a new _id

Mechanism. The read is served by a secondary that has not yet applied the debit. w:"majority" means the primary waited until a majority of voting members acknowledged the write; it does not mean every secondary has applied it. With six seconds of lag, the balance read at T+0 is a pre-debit value. And readConcern:"local" on a secondary returns whatever that node currently has, including data that can later be rolled back. The check therefore passes twice.

Is readConcern:"majority" alone enough? No, for two separate reasons.

1. Read-your-own-writes on a secondary additionally requires causal consistency: a session with causalConsistency: true (driver default) and readConcern:"majority", so the driver attaches afterClusterTime and waits for the secondary to reach the cluster time of the prior write.
2. More fundamentally, check-then-write is a TOCTOU race that survives even a perfect read on the primary. Two concurrent withdrawals both read 100 and both write -60. Read concern cannot fix that; atomicity can.

Smallest correct change. Do the affordance check and the debit in one conditional single-document update on the primary:

db.accounts.updateOne(
  {_id: acct, balance: {$gte: 60}},
  {$inc: {balance: -60}})

and treat modifiedCount == 0 as insufficient funds. One document, one atomic write, no transaction, no read concern tuning needed.

Only if the check must be reported separately from the debit should you use a session transaction with readConcern:"snapshot" and writeConcern:"majority"; the snapshot plus write-conflict detection prevents the double spend, at roughly double the latency.

The first write is majority-committed, the reply does not come back inside 10 seconds (a client-side socket timeout, not necessarily a retryable server error), the application retries, and the retry builds a new document because it generates a new _id. Two debits for one request.

What retryable writes cover. Retryable writes (default since 4.2) retry a single write command once when the driver sees a specific retryable error code, and the server uses the session id plus transaction number to make that replay idempotent. They cover exactly one case: the same write from the same session being replayed by the driver. They do nothing for an application-level retry that constructs a new command with a new _id.

Fix: make the operation idempotent instead of hoping the driver handles it.

• Derive _id deterministically from the caller's idempotency key, or add a unique index on {requestId: 1} in a processed collection and write the marker in the same document as the ledger entry so the write stays single-document atomic.
• On retry the insert fails with E11000 duplicate key; catch it and return the original outcome as success.
• Never generate a fresh _id per attempt, and make the retry carry the same requestId from the API layer - a gateway that mints a new requestId on retry defeats the whole scheme.
• Keep w:"majority" and keep retryable writes on; they reduce one class of duplicates but are not the idempotency mechanism.

No for a ledger. {w:1, j:true} waits for journaling on the primary only. It survives a mongod crash, but it does not survive a primary failover: the write may not have replicated, and a newly elected primary can roll it back. w:"majority" is the acknowledgment that makes a committed debit survive election. For money movement the latency cost is the price of the guarantee.

Journaling default. writeConcernMajorityJournalDefault (default true) means w:"majority" also implies journaling on a majority of members. Setting it to false makes majority acknowledgment non-durable under power loss, so do not touch it.

wtimeout semantics. If wtimeout expires, the write is not rolled back - it may still be applied later. The error means "outcome unknown", not "failed". Treat it like the socket timeout: retry with the same idempotency key so a duplicate is impossible, and never retry with a new _id.

Practical tuning without weakening the guarantee.

• Keep w:"majority" (no explicit wtimeout, or a generous one) and reduce the round-trip cost by batching debits per request where the business allows it.
• Use writeConcern on the write path only; send reporting reads to secondaries - those are the ones that tolerate lag.
• If you truly need lower write latency for a subset of data, isolate that data in a collection with a weaker documented policy - do not weaken the accounts collection.