Node.js Service That Leaks Memory Every 26 Hours

There are three interacting defects, and only one of them is a classic leak.

1. Unbounded memoisation (templateCache). The key includes templateVersion. Every content deploy or template edit creates a new key set for every tenant and locale combination, and nothing ever removes the old keys. The cache therefore grows monotonically with (tenants x locales x versions ever served). This is not a leak in the pointer sense - the map still references everything, so the GC correctly considers it live. Growth is a straight function of deploy count and tenant count, which is why the restart interval shrinks as the customer base grows.

2. Promise.race does not cancel the loser. When the 3000 ms timer wins, render(payload) keeps executing. It still holds the request context, the tenant, the recipient list, and its output buffers until it finishes or errors. Before the change, a slow render was bounded by the platform's HTTP timeout and the work eventually died with the socket. After the change, a slow render is orphaned: the client is answered at 3 s, and the orphan runs on for many more seconds, holding memory and CPU. So the fix reduced measured latency by moving work off the critical path rather than removing it. Under 8% traffic growth the number of simultaneously orphaned renders rises, so retained memory per unit time rises - which is exactly the compressed restart interval you observe.

3. Event-loop blocking amplifies both. JSON.parse on 12-30 MB payloads blocks the loop for hundreds of milliseconds. Every in-flight render is delayed during those stalls, which pushes more renders past 3 s, which creates more orphans. The three problems compound.

Diagnostic evidence to confirm before changing code: a heap snapshot diff between hour 1 and hour 12 showing retained growth under templateCache plus a rising count of pending promises; and an event-loop lag histogram that spikes when large webhooks arrive. Take the snapshots under production-like load, not on a quiet pod - idle snapshots will show you a healthy heap.

Priority 1 - bound the cache. Replace the Map with an LRU that has an explicit entry cap and TTL (for example 5,000 entries, 30 minutes). Better, split it: a small LRU for the current version per tenant and a version-keyed sub-map that is deleted on template change, so old versions are evicted by construction rather than by age. Expected effect: flat memory between deploys. Proof: run a 72-hour soak at 1.2x peak traffic, sampling RSS and heapUsed hourly, and export a gauge for cache entry count with an alert at 80% of cap.

Priority 2 - make rendering actually cancellable. Thread an AbortSignal through the render pipeline and have each await point check it; abandon on timeout rather than race. If the rendering library cannot be cancelled, run it in a worker thread and terminate the worker on timeout - termination is real cancellation. Remove Promise.race entirely so nobody reintroduces orphaning. Proof: add a render_in_flight gauge and a counter for abandoned renders; after the fix, in-flight count should track request rate with a bounded, non-growing queue, and abandoned renders should drop to near zero within one timeout window of a stall.

Priority 3 - stop blocking the loop on parsing. Move webhook parsing to a worker thread or a streaming parser, and reject payloads over a documented limit with 413 before buffering them. Proof: event-loop lag p99 should fall below 50 ms and stay flat during a burst test that sends 50 concurrent 25 MB payloads.

Priority 4 - make OOM impossible to reach silently. Set --max-old-space-size below the container limit (for example 1,300 MB for a 1.5 GB limit) so V8 pressure surfaces as GC and a heap alert instead of a SIGKILL, and remove the restart-on-OOM safety net from the deployment config, which currently masks the bug.

Ordering rationale: P1 and P2 reduce the growth rate; P3 removes amplification; P4 changes a silent failure into an observable one. Doing P3 first would look productive while the actual retention stayed.

The regression is a category error: a helper was used to bound latency while it only moved work off the request path. Three process controls catch that class of change, and they must be cheap enough to survive contact with a delivery deadline.

1. Treat latency changes as behavioural changes. A change to timeouts, retries, caching, or concurrency is never a one-line refactor; it goes in the change template with an explicit statement of what now happens to work that exceeds the limit - completed, cancelled, or abandoned. That one question would have surfaced the orphaning.

2. Soak tests in CI, not just load tests. A 30-minute load test runs long enough to show throughput and latency and too short to show a 26-hour growth curve. Add a nightly 4-hour soak at 1.2x peak in the pipeline, capturing heap and RSS trend, in-flight counters, and event-loop lag; fail the build on a positive slope above a set threshold. This is the only reliable detector for accumulation bugs.

3. A dependency and pattern rule for mutable module state. Add a lint rule that flags module-level Map, Set, [], and {} that are mutated after module load outside a designated cache module. It is crude and it will produce false positives, but each one forces a two-second conversation about eviction. Reviewers then have a mechanical prompt, not just memory.

Two supporting habits matter as much as the controls. First, make heap and in-flight metrics first-class next to latency and error rate on the service dashboard - the team celebrated p99 improving and never looked at a memory graph. Second, when a latency fix ships, record the mechanism in the pull request, not just the measurement. A number moving in the right direction is not evidence that the system became healthier, and the reviewer who asks what happened to the losing promise is the cheapest defence available.