Right-sizing a web tier after peak-hour latency spikes

Doubling every server is a scale-up decision, and it may be the wrong one for a stateless web tier. The first issue is cost: doubling vCPU on 24 servers is roughly a 100% compute increase, which finance has already rejected. The second issue is causality: p95 CPU at 88% during the peak does not prove that CPU is the bottleneck. Response time rose from 220 ms to 900 ms, but that could come from database contention, connection pool exhaustion, load balancer queuing, or a few slow requests.

Evidence required before deciding:

• Per-server CPU distribution during 19:00-21:00, not just the fleet average. If a few servers hit 95% while others sit at 40%, the problem may be uneven load balancing or long-lived connections.
• Database and connection-pool metrics during the same window. A stateless app that waits on a shared database will not improve merely by adding vCPU.
• Load balancer metrics: active connections, queued requests, and backend response time versus client response time. If the load balancer is queuing, the backend may not be the constraint.
• Request mix and latency by endpoint. A single expensive endpoint can raise p95 without requiring more CPU on every server.

If CPU is genuinely the constraint and the app is stateless, scale-out by adding more 4-vCPU servers is usually more granular and reversible than doubling all 24. If the app has a per-request CPU hotspot, optimize that code path first. Scale-up can still be appropriate for a small number of servers with a single-threaded or memory-bound workload, but the decision should follow measurement, not the peak CPU number alone.

Treat the golden image as the only approved starting point and move configuration into the image build, not into post-launch manual steps.

1. Version the image explicitly. Every image gets a version and a changelog. A server's image version should be visible from the instance metadata or a file written at build time.
2. Bake library and runtime configuration into the image. The 20 to 30 minutes of manual configuration is the drift source. Convert those steps into an automated build script that runs before the image is published.
3. Use a small, immutable launch path. At launch, the instance should receive only environment-specific values such as its role, private network address, and database endpoint. It should not receive library updates or hand-edited configuration.
4. Replace, do not patch, for routine changes. Roll out a new image version by adding new servers to the load balancer, waiting for health checks, then removing old servers. This also tests the image under real traffic.
5. Detect drift continuously. Compare a running server's package versions and key configuration files against the image manifest. Report drift rather than silently accepting it.
6. Keep a rollback image. Do not delete the previous two image versions until the new version has passed a full peak cycle.

The key judgement is that drift is a process defect, not a server defect. Rebuilding the image pipeline is cheaper than repeatedly fixing individual servers by hand.

The most likely cause is that the TCP health check on port 80 succeeds as soon as the port is open, which may happen before the application is actually ready to serve requests. During a deployment, a server can pass the check while its libraries, configuration, or database connection pool are still initializing. The load balancer then sends traffic to an unready server, requests fail, and the server is removed again. A second likely cause is that the manual configuration window creates an inconsistent state: the server is listening but returning errors because a library version or environment value is wrong.

Recommended changes:

• Use an application-level health check that exercises a readiness endpoint, not just a TCP handshake. The endpoint should verify that the app can reach its database and that required configuration is loaded.
• Add a grace period so a newly launched server is not judged during startup. Combine a startup grace period with a health-check interval that is shorter than the deployment window but not so short that normal garbage collection or a brief spike causes removal.
• Separate liveness from readiness. A server that is alive but not ready should be kept out of rotation without being terminated.
• Deploy in batches. Add one or two servers, wait for readiness, then remove the corresponding old servers. Do not replace the whole fleet at once.
• Check the deployment log against the image manifest before adding a server to the load balancer. If the manual configuration differs from the approved image, fail the deployment instead of sending traffic.

The design principle is that the load balancer should route on service readiness, and the deployment process should make readiness deterministic.