Scheduled Batch Spike on a Six-Instance Web Tier
TicketWave runs a six-instance web tier (4 vCPU, 8 GB each) in KR1 AZ1 behind an Application Load Balancer. Baseline CPU is 30%, and response time is stable. Every day at 23:00 a cron job starts a ticket-release batch on every web instance. The batch is single-threaded but memory-heavy: each process peaks at 7.5 GB RSS, and CPU rises to 95% for about 25 minutes. During that window, the load balancer returns 5xx to roughly 8% of requests. The Auto Scale group is min 6, max 8, with a scale-out policy of average CPU greater than 70% for 5 minutes and a 15-minute cooldown. It has never added an instance during the batch. The batch cannot be moved earlier, and the business will not accept a second nightly maintenance window. Month-end volume doubles the batch's memory footprint.
1. The Auto Scale policy looks correct on paper. Explain why it still fails to protect the 23:00 window, and list the evidence you would pull before changing anything.
Read the worked answer
Auto Scale is reactive, but the decisive problem is likely that the batch is baked into the instance template: every new instance also starts the 23:00 cron. Scale-out therefore adds more batch processes competing for the same database and memory, not clean web capacity. The 25-minute run is also shorter than boot plus application warm-up plus the 15-minute cooldown, so added capacity may arrive after the spike. Evidence to collect: the instance template and cron/systemd unit, Auto Scale activity history, per-instance CPU and memory around 23:00, load balancer target registration timestamps, and whether newly added instances show the batch process. Do not raise max or lower the threshold until you confirm the template. If the cron is in the image, scaling is actively harmful.
2. Propose a target architecture that isolates the batch and keeps the web tier responsive. Include instance family, scaling method, and the cost trade-off.
Read the worked answer
Move the batch off the web tier. Build a batch-only instance template with no web cron and a memory-optimized flavor such as 8 vCPU/32 GB, because a 7.5 GB peak on an 8 GB web instance is already close to swapping. Trigger it with a scheduler (a small scheduler instance or an Auto Scale scheduled policy) at 22:50, run the batch, write output to Object Storage or RDS, then terminate. Keep the web group fixed at 6; optionally schedule a small pre-warm to 7 at 22:45 only if traffic justifies it. Cost: one batch instance for about an hour per day plus storage, versus keeping max 8 web instances all day and still failing. This also prevents batch memory from evicting web processes.
3. What operational guardrail would prevent a repeat during month-end, when batch volume doubles?
Read the worked answer
Month-end doubles memory, so 32 GB may still be tight if the batch loads everything at once. Enforce chunking (for example, 50,000 tickets per chunk), a hard memory limit at 80% with alerting, and a pre-flight estimate from last month's row count. Add a monitoring alarm on batch RSS and duration; if RSS exceeds 24 GB, fail fast and page rather than letting the OS OOM killer decide. Keep the batch instance isolated so a failure cannot take down checkout. Also document that scaling the web group is not the mitigation for a batch-memory problem.