Connection Exhaustion and Alert Fatigue on a Shared M30

Do the pool arithmetic. Each warm Lambda container can open up to 100 sockets. With 40 functions at up to 200 concurrent executions, the theoretical ceiling is 800,000 connections; in practice just 32 concurrent containers holding full pools reach 3,200. The cluster is not under-resourced - the clients are misconfigured, and M40 only raises the ceiling while leaving the growth curve intact.

Fixes, in order of impact

1. maxPoolSize: 2-5 per function, chosen as expected queries-in-flight per container, not as a default. A Lambda that handles one request at a time needs a pool of a handful, not 100.
2. Reuse the client: instantiate outside the handler so containers share one MongoClient across invocations. Clients created per invocation leak pools until the container is recycled.
3. Set maxIdleTimeMS (e.g. 60000) and minPoolSize: 0 so idle warm containers stop holding sockets during low traffic.
4. Bound concurrency: reserved/provisioned concurrency per function caps containers, which caps connections.
5. Verify with the connections-per-host metric during a peak - expect a drop of an order of magnitude before touching tier size.

Why CPU matters here: 40-55% CPU during the failure means the cluster was healthy and rejecting handshakes, not saturated. That is evidence against the M40 request.

Principle: page on sustained customer impact; ticket on trends; delete everything that describes normal behaviour. Thresholds come from the measured baseline over 30 days, not from defaults.

Page-worthy (multi-condition, sustained)

• Connections above 80% of the ceiling for 10 minutes - this is the alert that should have existed before today's outage.
• Replication lag above 10 seconds for 5 minutes, and any primary election outside a maintenance window.
• Disk above 85% with projected-full under 30 days: calculate from the growth trend, not a flat 90% line that fires forever and gets muted. Muting a permanently-red alert is a process failure; delete or re-scope it.
• CPU above 90% for 15 minutes, scoped to the primary.

Ticket/Slack only: query targeting ratio drift, slow-query regressions, index suggestion backlog, connections above 60%.

Noise controls: route by project and owner so nine teams stop receiving each other's alerts; suppress during the documented nightly batch window; require an alert to have an owner and a runbook or it gets deleted; review the alert list monthly against incidents that actually happened.

Framing to the team lead: the M40 buys headroom at 3.5x cost and still pages you, because the alert conditions were never tuned to this workload's baseline.

Start at the query, not the hardware. Quiet-hour latency rules out contention, so a fixed cost per request dominates.

1. Performance Advisor: top namespaces and slow-query shapes for the checkout path over the last 24 hours.
2. explain("executionStats") on the top shape. Compare totalDocsExamined and totalKeysExamined to nReturned. A ratio far above 1 means a scan or a non-selective index; a ratio near 1 with high latency means the work is fine but the data is not in memory.
3. Look for in-memory sorts (SORT stage with spill) and blocking stages - a sort that exceeds the 100 MB limit fails, but a large non-blocking sort still costs.
4. Check whether the query runs on a secondary with readPreference and is paying replication lag or cache-cold reads; secondaries have their own working set.
5. Only then look at disk IOPS, page faults, and tier sizing.

Deliverable: a short list of candidate indexes with the exact query shapes they serve, applied one at a time with before/after p99 from the same traffic window, plus a note of the write cost each index adds - the cluster writes 2,400/s and index sprawl hurts the batch window.

Do not open a support case for 'slow cluster' before step 2; it will be routed back with a request for an explain plan.