Memory leak three weeks before peak
Orion Retail's order service runs in 12 containers. Memory grows about 4% per hour and each container is OOM-killed and restarted roughly every 30 hours; each restart produces a 90-second spike in errors. Black Friday is three weeks away with 5x expected traffic. The autoscaler scales on CPU, which stays low because the growth is in memory. Option A: raise the memory limit and add a nightly 03:00 restart. Option B: rewrite the caching layer - six weeks of one team's capacity, and that team also owns the promotional pricing feature Black Friday depends on. Option C: switch autoscaling to a memory metric; the platform team warns of thrashing and roughly three times the node cost.
1. Choose between A, B and C, and justify the choice against the date.
Read the worked answer
Rule out the options that fail on their own terms
- C is wrong on principle, not just cost. The memory metric is measuring the leak. Scaling on it treats the symptom as demand, multiplies node cost, and the thrash adds restart churn on top of the spikes you are trying to remove. It also makes the failure mode worse under peak, when you can least afford it.
- B is the correct long-term fix and the wrong thing to start now. Six weeks of a team that also owns the feature peak revenue depends on means it cannot land before Black Friday. Scheduling it now guarantees either a half-finished rewrite at peak or a missed pricing launch.
- A is the only option that holds through the date, provided the restart is made graceful - drain connections, rolling restart, or restart one container at a time - so the 90-second spike becomes a non-event rather than something you have scheduled deliberately.
Recommendation
A now, with the restart automated, monitored and documented, plus a raised memory limit sized for peak growth rate rather than today's. Book B for January with the caching team, and reject C outright. Choosing the smallest blast radius that survives the deadline is not settling; trading a revenue-critical launch for an unfinished rewrite is the worse trade.
2. What would change your mind about option A?
Read the worked answer
The central assumption in A is that a nightly restart fits comfortably inside the growth cycle. That assumption is load-dependent, and the scenario gives you the reason:
At 4% per hour the container survives about 30 hours. At 5x traffic the growth rate is very unlikely to stay at 4%/hour - leaks of this kind scale with request volume. If the interval drops to 10–12 hours, a 03:00 restart no longer gets you to the next 03:00, and you are OOM-killed during trading hours, which is exactly the outcome A was supposed to prevent.
So the conditions that overturn A:
- Measured growth rate under load test at or near 5x proves the safe interval is shorter than 24 hours.
- The restart cannot be made graceful within three weeks (rolling restart blocked by session state, for example).
- Node capacity at 5x means restarts take materially longer under load than they do today.
If any of those hold, A collapses and the answer becomes a short-lived horizontal fix - more replicas with a shorter, deliberately staggered recycle - not C. The test to run this week is a load test that measures memory growth per request, not a guess at 4%.
3. How do you know afterwards whether the decision was right?
Read the worked answer
Set the measures before peak, or you will rationalise whatever happened.
- Customer-facing: error rate and p95 latency during peak windows, compared with a pre-agreed threshold - not compared with last year, which had different traffic.
- Stability: OOM kills during the peak period (target zero), restart count, and whether any restart landed inside trading hours.
- Mechanism: measured memory growth rate versus the rate you predicted. If growth was far faster than modelled, the decision survived on luck and that belongs in the review.
- Cost: node spend during peak versus forecast, which is also the check that you did not quietly drift into option C.
- Delivery: whether the promotional pricing feature shipped - because protecting that was half the reason for the choice.
Hold a review one week after peak while the data is fresh, and confirm the January rewrite is still booked. A mitigation that works becomes permanent by default; the review is what stops that.