Scaling a Ticket Sale Without Rewriting
A ticketing platform built on ODC has a major on-sale in three weeks. The event page uses one aggregate with six joins over a 3-million-row events table and a 12-million-row availability table. A client action calls 20 server actions serially to render seat categories, price tiers, and fees. At 500 concurrent users, p95 is 8 seconds; at projected 50,000 concurrent users, the team expects failure. Database CPU is already 95%, and the connection pool is exhausted during load tests. Autoscaling adds app instances, but the database remains the bottleneck. The checkout flow writes an order, decrements inventory, and calls a payment provider synchronously. The team proposes a last-minute rewrite to microservices. The CTO wants a plan that protects the on-sale, with rollback and load-test evidence.
1. Prioritize performance fixes for the on-sale. Explain expected impact and risks for each.
Read the worked answer
Prioritize by removing database work and serial round trips before changing architecture. First, replace the 20 serial server actions with one or two server actions that return a single DTO; the current pattern multiplies latency and connection use. Second, add a read model or cache for event page data with a short TTL and explicit invalidation on inventory change; 50,000 users should not each run six joins. Third, paginate or lazy-load seat categories and precompute fees. Fourth, add covering indexes for the availability query and verify the execution plan; 95% CPU suggests missing indexes or repeated scans. Fifth, make checkout asynchronous for payment confirmation while keeping inventory decrement transactional and idempotent.
Risks: stale cache can show unavailable seats, so pair with optimistic reservation; async payment changes UX and requires status messaging. A microservices rewrite in three weeks adds risk and does not remove the database bottleneck. Load-test each change at 2x projected peak.
2. The team proposes adding more app instances. Evaluate this and explain why it may worsen the incident.
Read the worked answer
Adding app instances increases compute parallelism but also multiplies database connections. If each instance opens a fixed pool and the pool is already exhausted, autoscaling can turn a slow system into a connection storm. The database is at 95% CPU, so more concurrent queries will increase contention, lock waits, and latency. More instances may help if the bottleneck is app CPU or network, but the evidence points to the database.
I would cap total connections with a shared pooler or per-instance budget, set autoscaling on queue depth and p95 rather than CPU, and add admission control or rate limiting at the edge. Scale reads with a read replica only for tolerant queries, and keep inventory decrements on the primary. Add a circuit breaker so checkout fails fast rather than holding connections. The decision test is whether p95 improves at constant database load; if not, adding instances is harmful. Fix the query and serial-call pattern first, then scale with measured headroom.
3. Define the lifecycle, deployment, and rollback plan for the on-sale release.
Read the worked answer
Treat the on-sale as a controlled release, not a normal weekly deploy. Freeze schema changes except backward-compatible additions; any index or column change must be online and reversible. Load-test in staging at 2x the projected peak with production-like data volumes and the same connection limits. Deploy during a low-traffic window at least 72 hours before the on-sale, then run a canary or feature-flagged path for the new read model and checkout queue. Keep the old event page path available behind a flag.
Rollback criteria: p95 above 3 seconds, error rate above 1%, queue age above 60 seconds, or database CPU above 90% for five minutes. Rollback must not require a data restore; use flags and backward-compatible contracts. Monitor database CPU, connections, p95, order success, payment failures, and inventory discrepancy. Run a game day with the payment provider sandbox. After the event, remove the old path and review the incident metrics. This plan buys evidence and reversibility rather than promising a rewrite.