Helpdesk SaaS: the 09:00 latency cliff and the instinct to add servers
A helpdesk SaaS product serves about 400 concurrent agents. Between 08:55 and 09:05 the p95 API latency climbs from 0.8 s to 4.2 s and some logins time out. The architecture is an Application Load Balancer in front of one EC2 Auto Scaling group. That same group serves the React bundle, product images and the JSON API from the application servers. Sessions live in the JVM heap of whichever instance handled the login, and the target group has sticky sessions enabled. The database is a single db.r6g.large in one Availability Zone, peaking at 85 percent CPU during the morning window. The scaling policy triggers on average CPU across the group, which sits near 30 percent because static-asset traffic is cheap to serve. Agents are complaining publicly and the CTO wants to "just add more servers".
1. Why will adding servers probably not fix the 09:00 latency, and in what order would you fix it instead?
Read the worked answer
Three things conspire against "more servers". The scaling metric is diluted: average CPU stays near 30 percent because static traffic is cheap, so the group never scales when the API is actually struggling. The database is a shared bottleneck at 85 percent CPU, so extra application servers push more queries at the one component that cannot absorb them. And sticky sessions mean new instances do not evenly relieve existing ones.
Order of work, cheapest and highest leverage first:
- Move the React bundle and images to object storage behind a CDN. This removes the traffic that is polluting the scaling signal and cuts load on the application tier at the same time.
- Move sessions to an external store (ElastiCache or DynamoDB) and disable stickiness, so instances become interchangeable.
- Change the scaling policy to a request-count or target-tracking metric that reflects API concurrency, and add a scheduled scale-out ahead of the 08:55 ramp, because reactive scaling always lags a step change.
- Only then address the database: caching hot reads, read replicas, or a larger instance, in that order of cost.
Doing step 4 first would mask the real problem and cost the most.
2. What is wrong with sticky sessions here, and what does removing them require?
Read the worked answer
Sticky sessions are being used to compensate for the application holding session state in the JVM heap. That is a design smell with two consequences.
First, load does not spread. A few long-lived agent sessions pin to one instance while others idle, so the group's average utilisation looks healthy while individual instances saturate - which is exactly why average CPU near 30 percent coexists with 4.2 s p95 latency.
Second, stickiness does not even deliver its promise during failure: if the pinned instance is replaced by a scale-in event or an unhealthy host, the agent loses the session and re-authenticates anyway. You pay the load-balancing cost without buying real resilience.
Removing stickiness requires externalising session state first, not disabling the setting and hoping. Move sessions into a shared store, make the application read and write session data there, verify a login survives an instance termination test, and only then turn stickiness off. Sequence matters: disabling stickiness before state is externalised turns a performance problem into a logout storm at 09:00.
3. The team proposes Multi-AZ for the database as the fix for the morning slowdown. What does that actually solve and what does it not?
Read the worked answer
Multi-AZ addresses availability, not throughput.
What it solves: if the Availability Zone hosting the database fails, or the instance itself fails, the service fails over to a standby in another AZ, usually within a few minutes, and DNS is updated. It also lets AWS apply maintenance without a long outage. That is a genuine and necessary fix for a single-AZ database, and it should be done.
What it does not solve: the morning CPU spike. With a Multi-AZ DB instance deployment the standby is not readable, so it takes none of the query load; the primary still does all the work at 85 percent CPU. Adding application servers through Multi-AZ changes nothing either.
So say both things: Multi-AZ is the right answer to "one AZ is a single point of failure", and it is the wrong answer to "our queries saturate the primary every morning". The throughput work is caching hot reads, moving reporting queries to read replicas, and reducing the queries each login issues - then Multi-AZ is the durability layer underneath, not the performance fix.