Serverless fan-out and a read that cannot see its own write

The math is the diagnosis. A MongoClient is a pool, not a connection: maxPoolSize defaults to 100 per server. Two hundred concurrent functions each constructing their own client can, in the worst case, ask for 200 × 100 = 20,000 connections against a cluster whose ceiling is around 1,500. The cluster refuses or queues, the driver's server selection times out, and every function retries, which makes it worse. The 400 ms latency is largely queueing, not slow queries.

What to ship:

• One client per process, created at module scope and reused across invocations (let client; if (!client) client = await connect()), so warm Lambdas share a pool instead of building one per request. Never call close() per request.
• Size the pool to the actual concurrency per instance, not to the default. For a Lambda that handles one request at a time, maxPoolSize: 5 (with minPoolSize: 0) is generous. Total expected connections ≈ concurrent functions × maxPoolSize; keep that comfortably under the tier limit.
• Add maxIdleTimeMS so idle sockets are reaped, and set serverSelectionTimeoutMS and connectTimeoutMS to values that fail fast rather than hang a Lambda until its own timeout.
• If the tier cannot be raised, use a connection proxy or a driver-level pooling layer rather than opening connections from every function.

Do not solve this by raising maxPoolSize. That multiplies the problem.

secondaryPreferred routes the read to a secondary. Replication is asynchronous, so at the moment of the read the secondary may not yet have applied the insert - a read-your-writes violation, not data loss. The order exists; that node has not heard about it. It is more likely to bite right after a write and it is worse under write load, which matches the intermittent report.

Fixes:

1. Read your own writes from the primary: use readPreference: 'primary' for the confirmation path (or, with the Node driver's findOne on a collection obtained from a primary-preferring client). Simplest and correct.
2. Use causal consistency: run the write and the read in the same session with causalConsistency: true (the default for most driver sessions). The read then carries afterClusterTime, so the secondary will not answer until it has caught up to the write's cluster time. This keeps secondary reads (useful when reads dominate) while guaranteeing read-your-writes per session. It does not help across sessions or across different clients unless the session is propagated.
3. Use readPreference: 'primaryPreferred' if you want secondary reads only during primary unavailability, accepting that reads may briefly go stale.

What you should not do is add a sleep or a retry loop to paper over replication lag. That trades a deterministic bug for a probabilistic one.

First, distinguish two mechanisms that are often confused. Retryable writes (retryWrites: true, the default) let the driver transparently retry a single write once if it fails with a retryable error such as a network blip or a stepdown. The driver tags the retried operation with a transaction number and the server deduplicates it, so the insert is applied at most once. That protection covers one automatic retry inside the driver - it does not cover the application retrying a whole request.

The duplicate orders come from application-level retry. If the payment endpoint is retried by the caller (or by the Lambda/API gateway on timeout), the handler runs again and executes a fresh insertOne, which the server has no way to recognise as the same logical operation. The write concern also matters: with the default w: 1, a timeout does not tell you whether the write applied, so the safe-looking retry is genuinely unsafe.

The fix is idempotency at the data layer: generate a request identifier (orderRef) on the client before the first attempt, put a unique index on it, and make the insert the idempotency gate. On a duplicate-key error, fetch and return the existing order instead of inserting. That makes retries free regardless of how many layers retry.

Also reconsider the write concern on the order insert: {w: "majority"} means an acknowledgement survives a failover, and for a payment path the latency is worth it. And keep transactions out of it unless the order spans multiple documents - a single-document insert is already atomic and does not need one.