An EU-Only Latency Regression and a Premature Cache
An e-commerce platform runs a public API at 200 requests per second. Before release 12.4, p95 latency was 180 ms and database CPU averaged 45%. After the release, EU users see p95 of 950 ms while US users remain at 190 ms; database CPU is now 85% and the error rate has risen from 0.05% to 0.3%. The release added a product-recommendation endpoint that calls the catalogue service once per returned item. In response, the team deployed a Redis cache in front of the catalogue with a 300 s TTL. The cache hit ratio is 38%, EU p95 has fallen only to 780 ms, and the support team now receives about 14 stale-price tickets per day. A proposal on the table is to add two more database read replicas and raise the cache TTL to 3600 s. The EU cluster also received a configuration change to its connection-pool limit in the same release.
1. Identify the most probable root cause of the EU-only latency regression and explain why adding read replicas is or is not the right first response.
Read the worked answer
The strongest signal is the new endpoint that calls the catalogue once per returned item. That is an N+1 fan-out pattern: one client request becomes one query per item. If EU responses return more items, or if the EU connection-pool change reduced concurrency, the same code produces far more database work in EU than in the US. The region-specific pattern, the timing, and the rise in database CPU from 45% to 85% all fit this explanation better than a general capacity shortfall.
Read replicas add read capacity but do not reduce the number of queries. With N+1 fan-out, replicas may temporarily lower CPU on the primary while multiplying network round trips, replication lag, and cost. They also do not fix the 0.3% error rate if errors come from pool exhaustion or timeouts.
The first response should be diagnostic: trace one recommendation request and count downstream catalogue calls; compare EU and US item counts; inspect the new pool limit against request concurrency; and check whether the endpoint batches or loops. If the call count is the problem, the fix is batching, pagination, or a single bulk catalogue lookup, not more replicas. Replicas become reasonable only after the query count per request is bounded.
2. Assess the Redis cache design. Use the 38% hit ratio and the stale-price tickets to argue whether the cache is currently a net benefit.
Read the worked answer
A 38% hit ratio means 62% of lookups still reach the catalogue. The cache removes less than half of the added query load, which is consistent with p95 falling only from 950 ms to 780 ms. The 300 s TTL also guarantees that a price can be wrong for up to five minutes, and the 14 stale-price tickets per day show that this is a real business cost, not a theoretical one.
There are further risks. With a fixed TTL, entries expire together and can cause a stampede that briefly restores the original load. Caching item-level data without invalidation on write makes correctness depend on time rather than on the event that changed the data.
So the cache is not clearly a net benefit yet. It adds a consistency failure mode and operational complexity while removing under half the database traffic. A defensible cache would use write-through or event-based invalidation for prices, per-item TTLs, request coalescing, and monitoring of hit ratio and database QPS reduction. If the hit ratio cannot be raised above roughly 70% and stale tickets cannot be reduced to near zero, the better move is to fix the N+1 query and remove the cache rather than raise the TTL to 3600 s, which would make staleness worse.
3. Recommend a decision between rollback, fix-forward, and scaling, with specific thresholds.
Read the worked answer
Treat the three options as conditional, not as a preference.
Rollback if any of these hold for 15 minutes: p95 above 1,000 ms in EU, error rate above 0.5%, or database CPU above 90% with queueing. Rollback should target the recommendation endpoint first, via a feature flag, because that is the suspected cause; a full release rollback also reverts the connection-pool change and is the safer choice if the flag cannot isolate the endpoint.
Fix-forward only after the query count per request is measured and a canary at 5% of EU traffic shows p95 below 250 ms and error rate below 0.1% for at least 30 minutes. The fix must batch catalogue calls and restore a pool limit consistent with concurrency.
Scaling with read replicas or more nodes is justified only after query fan-out is bounded and a load test shows the primary is still the bottleneck. Scaling before that hides the defect and raises cost.
On the cache, keep it only if a revised design achieves at least 70% hit ratio with event-based invalidation and fewer than two stale-price tickets per day; otherwise remove it. Record the decision and thresholds in the incident review so the same regression is caught by the canary next time.