Latency Spike After Adding a Second Region
A media site serves user-generated images from a single OSS bucket in the Singapore region. Users are concentrated in Singapore and in Frankfurt. Frankfurt users report 1.5–2.5 second image load times while Singapore users see under 300 ms. The team's first instinct is to add a second OSS bucket in Frankfurt and have the application choose a bucket based on the user's location. They are worried about consistency: a user who uploads in Frankfurt and then travels to Singapore must still see their image. The site also uses a CDN in front of OSS. Traffic is read-heavy (95% reads, 5% uploads). Uploads are small and infrequent.
1. Diagnose the latency and evaluate whether a second regional bucket is the right fix.
Read the worked answer
The latency is cross-region distance: Frankfurt users are fetching from Singapore, and the round trip plus any cache misses dominates. A second bucket in Frankfurt would reduce origin latency, but it introduces a consistency problem: an object uploaded to the Frankfurt bucket is not automatically in the Singapore bucket, so a traveling user could miss their image. The application-level bucket selection also adds branching logic and failure modes.
Given 95% reads, the higher-leverage fix is the CDN, not a second origin. If the CDN is already present, the question is why Frankfurt users are hitting the origin at all - likely cache misses, short TTLs, or uncached object paths. Fixing cache behavior usually removes most of the latency without any consistency trade-off.
2. Propose a solution that reduces Frankfurt latency while preserving read-after-write visibility for traveling users.
Read the worked answer
Recommended layered approach:
- Tune the CDN first. Increase TTLs for immutable image paths, enable prefetch/warm-up for popular objects, and confirm the CDN has points of presence serving Frankfurt. This addresses the 95% read path with no consistency risk.
- Keep a single authoritative origin (Singapore) so writes are immediately visible everywhere. Read-after-write is preserved because there is only one source of truth.
- If origin latency still matters for cold objects, add a caching layer or origin shield closer to Frankfurt rather than a full second writable bucket. A read-through cache avoids the split-brain problem.
- Only if writes must be local would you consider multi-region replication, and then you must design for replication lag (e.g., read-your-writes routing or version checks). For 5% small uploads, that complexity is not justified.
So: fix the CDN, keep one origin, avoid the second writable bucket.
3. What would change your recommendation toward multi-region storage?
Read the worked answer
I would reconsider if: uploads became a large share of traffic or latency-sensitive for local users; regulatory or data-residency rules required data stored in-region; or availability requirements demanded surviving a full regional outage. In those cases, multi-region replication with an explicit consistency model (replication lag tolerance, read-your-writes routing, or conflict resolution for concurrent writes) becomes necessary. The key is that multi-region storage is a consistency and residency decision, not primarily a latency decision - latency is better solved by caching.