Node.js on Lambda: connection storm and a doubly applied $merge

Diagnosis. The connection count is concurrent invocations × connections per client, and the driver's default maxPoolSize is 100. 200 concurrent lambdas can therefore open up to 20,000 sockets - even though a pool only opens on demand, the churn is the killer: every invocation does a fresh TLS handshake plus hello, uses one connection, and closes the client, so the pool is destroyed before it is ever reused. Server selection then has to fight through a connection-establishment storm and gives up at the default serverSelectionTimeoutMS of 30,000 ms, which is exactly the error reported.

Fixed lifecycle:

const client = new MongoClient(uri, {
  maxPoolSize: 10,
  minPoolSize: 0,
  maxIdleTimeMS: 60000,
  serverSelectionTimeoutMS: 5000,
  appName: 'daily-stats-job'
});
let connected;
export const handler = async (event) => {
  if (!connected) connected = client.connect();
  await connected;
  // ... work ... never call client.close() here
};

Reasoning per setting:

• Module scope: the client is created once per container and reused across warm invocations; only cold starts pay the ~300 ms handshake.
• No close() in the handler: closing destroys the pool and forces the next invocation to reconnect - the direct cause of the churn.
• maxPoolSize: 10: a Lambda container handles one request at a time, so 10 sockets is already generous. Budget globally: maxPoolSize = allowed_connections / max_concurrent_containers (e.g. 3,000 allowed / 300 containers ≈ 10).
• minPoolSize: 0: idle containers should hold no sockets.
• maxIdleTimeMS: 60000: reaps sockets that went stale while the container was frozen.
• serverSelectionTimeoutMS: 5000: fail fast instead of hanging 30 s and burning the invocation.

Also remember the pool is per MongoClient per process: two clients in one process is two pools.

Because pool exhaustion is not the bottleneck - connection establishment is. The error is MongoServerSelectionError, meaning the client could not complete server selection, not MongoWaitQueueTimeoutError, which is what an exhausted pool produces. Raising maxPoolSize targets the wrong failure.

What it actually does:

1. Raises the ceiling on sockets. maxPoolSize: 200 × 200 concurrent containers = 40,000 potential connections against a mongod/Atlas tier that has a maxIncomingConnections limit and a per-connection memory cost. You convert a slow failure into a hard refusal.
2. Amplifies the storm. Each new client still opens its own connections and does its own handshake; a bigger permitted pool lets each client open more before backing off, so the primary's CPU goes into authentication and handshake work instead of queries.
3. Adds latency, not removes it. A pool of 200 in a process that runs one operation at a time leaves ~199 idle sockets to keep alive and to time out; the per-invocation work does not get faster.
4. Hides the real design error. The client is still being created and closed per invocation, so the pool is never reused regardless of its size.

The rule to state on the exam: maxPoolSize is a per-client cap, not a per-cluster budget. Fix the lifecycle first, then size the pool from the global connection budget. If queueing ever becomes the real issue, you would see MongoWaitQueueTimeoutError on a reused client, and that is the case where a modest increase is justified.

Why retryWrites does not help. $merge is a write stage inside an aggregation, and aggregations are not retryable writes - the driver will not replay them and the server has no session/transaction number to deduplicate against. When Lambda retries (its at-least-once delivery on timeout), the whole pipeline runs again.

Why the numbers double. Re-running an accumulating whenMatched pipeline adds twice:

// doubles on re-run
$merge: { into: 'daily_stats', on: ['date','metric'],
          whenMatched: [{ $set: { total: { $add: ['$total', '$$new.total'] } } }] }

Any $add/$sum against the existing document is not idempotent.

Make it idempotent by recomputing, not accumulating:

$merge: { into: 'daily_stats', on: ['date','metric'],
          whenMatched: 'replace', whenNotMatched: 'insert' }

with the pipeline computing the full statistic from the source collection for the target date range ($match, then $group). Re-running produces a byte-identical document, so duplicate delivery is harmless.

If accumulation is genuinely required (e.g. incremental adds), gate it with a claim: a jobs collection with _id = 'daily_stats:2024-06-01', and

const claim = await jobs.findOneAndUpdate(
  {_id: jobId, status: 'pending'},
  {$set: {status: 'running', startedAt: new Date()}},
  {returnDocument: 'after'});
if (!claim) return; // already done or in flight

The unique _id makes the claim atomic, so a duplicate invocation finds nothing to claim. Pair the claim with a TTL or explicit completion marker so a genuine failure can be retried. And note the earlier fix still applies: even a correct $merge should run on a reused client, not a per-invocation one.