TicketHarbor: Surviving a Ten-Minute Traffic Wall
TicketHarbor sells concert tickets. Normal traffic is 400 requests per second, but an on-sale event produces 20,000 requests per second for about ten minutes, then drops. The current stack is one m5.2xlarge web server and a self-managed MySQL instance in a single Availability Zone. Last month's on-sale drove CPU to 100%, the site returned 503s for six minutes, and the database crashed; the company lost an estimated $180,000 in sales. The engineering team has four people, a fixed annual budget, and no appetite for managing Kubernetes. Static seat maps and artist images are served from the web server and account for 70% of bytes. The CEO wants no more downtime before the next on-sale in eight weeks.
1. Redesign the web tier and content delivery. Explain how each component addresses the observed failure.
Read the worked answer
Load balancing: put an Application Load Balancer in front of the web tier across three Availability Zones. The single server was both a capacity and availability failure point.
Auto Scaling: run an Auto Scaling group with a minimum of three instances and target-tracking policies on CPU and request count. Because a ten-minute spike outruns reactive scaling, add a scheduled action 30 minutes before each on-sale, plus a warm pool of pre-initialized instances.
Static content: move seat maps and artist images to S3 behind CloudFront. If 70% of bytes leave the origin, caching them absorbs most traffic and reduces the number of web servers needed. Set cache-control headers so images are not re-fetched per user.
Protection: add AWS WAF rate-based rules to blunt bots and scrapers that inflate the 20,000 requests per second. A queue or virtual waiting room can smooth the spike if fairness matters.
Operations: Route 53 health checks and CloudWatch alarms, plus a pre-event checklist. With four engineers, managed services are appropriate; self-managed Kubernetes would add operational load the team cannot absorb.
2. The database crashed. Should the team choose Multi-AZ or a read replica? Justify the decision.
Read the worked answer
Choose Multi-AZ first. A Multi-AZ deployment maintains a synchronous standby in another Availability Zone and performs automatic failover to it, usually within a minute or two, using the same endpoint. The failure here was a database crash in a single AZ; Multi-AZ directly addresses that availability gap.
A read replica is not a failover mechanism. Replication is asynchronous, so a replica can lag and lose recent writes if promoted, and promotion is a manual decision. Read replicas exist to scale read traffic, not to guarantee recovery.
That said, the two solve different problems and can be combined. If the on-sale is read-heavy, add read replicas after Multi-AZ is in place and point reporting or catalog reads at them. Also address connection storms: 20,000 requests per second from many short-lived web connections can exhaust database connections even when CPU is fine. Use RDS Proxy to pool and throttle connections.
Order of work: Multi-AZ for failover, RDS Proxy for connection pressure, then read replicas for read scale if metrics show they are needed.
3. How would you handle the ten-minute spike cost-effectively without buying year-round peak capacity?
Read the worked answer
Separate capacity that must exist all year from capacity that exists for ten minutes. The baseline of three web instances runs continuously; the spike is handled by scheduled scaling, a warm pool, and CloudFront caching rather than by permanently running dozens of servers.
Concretely: schedule the Auto Scaling group to a higher desired capacity 30 minutes before the event, let target tracking add instances during it, and scale back in afterward. Warm pools keep pre-initialized instances ready so boot time does not consume the spike. CloudFront and S3 absorb static bytes, which cuts the number of origin instances required. WAF rate limiting prevents bot traffic from consuming the capacity you paid for.
If the application can be adapted, a waiting room that queues users is cheaper than serving every request simultaneously and protects the database.
Avoid commitments sized for the spike: a one-year Savings Plan or Reserved Instance purchases should cover only the steady baseline. Peak capacity should be On-Demand or, for interruption-tolerant background work, Spot. Review the event's cost and utilization afterward so the next on-sale is sized from data.