MongoDB Atlas Admin Associate: Hot Working Set on a Single Replica Set: Scale Up or Shard?

Hot Working Set on a Single Replica Set: Scale Up or Shard?

Your SaaS platform runs one 3-node replica set on Atlas M40 (16 GB RAM, 4 vCPU per node) in AWS us-east-1. The dataset is 260 GB; the hot working set is roughly 95 GB and growing about 8 GB per month. Peaks run 9,000 reads/s and 2,400 writes/s for three hours, twice daily. During peaks you see 1,200-1,800 page faults per second, dashboard p99 of 380 ms, and CPU at 78-85%; overnight page faults fall to near zero and p99 drops below 40 ms. About 70% of reads hit an append-only events collection, always filtered by tenantId equality plus a createdAt range. That collection has no index other than _id, and each tenant's documents are written in time order. The largest tenant holds 22% of all documents. Product wants dashboard p99 under 100 ms at peak. Finance approved either an M80 upgrade (roughly 8x the RAM, about 3.5x the monthly cost) or a sharded tier, but not both. DR requires a read-only copy in eu-west-1 with a 15-minute RPO.

1. Recommend the M80 upgrade or sharding, justifying the choice from these numbers. What would you do first if neither purchase is approved this quarter?

Read the worked answer

Reframe the symptom first. Page faults mean the storage engine is going to disk for documents it will later discard. The setup says the dominant query filters tenantId equality plus a createdAt range, but the only index is _id. That is a collection scan per query: the engine faults in far more pages than any working set requires, and CPU burns on filtering. Both tiers and sharding would be expensive ways to hide a missing index.

Order of operations

Build {tenantId: 1, createdAt: 1} on events and re-measure page faults, CPU, and p99 over a full 24-hour cycle. Docs examined per doc returned should approach 1 in the slow-query log.
Only then ask whether the resident hot set (index plus repeatedly read documents) still exceeds 16 GB. A 95 GB hot set on 16 GB is a real memory deficit; on M80 it fits with headroom, so vertical scale wins on cost and operational simplicity.
Sharding pays off when the limit is write throughput, per-node IOPS/storage, or geo-placement - not here: 2,400 writes/s and 260 GB are modest, and sharding does not exempt any shard from holding its tenants' hot data.

If budget is frozen: keep the index, then shrink the hot set - TTL/Online Archive on old events documents, pre-aggregated per-tenant dashboard summaries, and covered queries so the dashboard never touches the documents.

2. Assume sharding becomes necessary next year. Choose a shard key for `events` and explain the trade-offs against the alternatives a colleague might propose.

Read the worked answer

Chosen key: {tenantId: 1, createdAt: 1} (ranged). Every dashboard query carries an equality on tenantId and a range on createdAt, so the mongos can target exactly one shard and the shard's index serves the range locally. Cardinality is high (many tenants), and the largest tenant at 22% still splits across chunks along createdAt, so chunks stay under the size limit.

Alternatives and why they lose

{tenantId: "hashed"} spreads writes perfectly but destroys range locality: a createdAt range inside a tenant becomes a scan of all that tenant's documents on the shard. Bad for a read-dominated workload.
{createdAt: 1} alone is monotonically increasing, so all inserts land on the max chunk - a single hot shard.
{_id: "hashed"} makes every dashboard query scatter-gather across all shards.

Risks to monitor: a very large tenant can concentrate IOPS on one shard; watch chunk distribution and per-shard CPU. Choose once - resharding 260 GB later works but costs a full data rewrite and a maintenance window.

Coupling to DR: if you later zone-shard for eu-west-1, the zone key (region) must be in the shard key prefix, e.g. {region: 1, tenantId: 1, createdAt: 1}. Decide that now, because it changes this answer.

3. How would you deliver the eu-west-1 read-only copy with a 15-minute RPO, and what does that decision cost you?

Read the worked answer

Options considered

A separate Atlas cluster in eu-west-1 kept current by a change-stream replication job: cheapest on paper, but it is application code you now own, with resume-token, ordering, and duplicate-apply bugs, and 15-minute RPO is easy to breach silently.
Backup restore into eu-west-1: RPO equals the snapshot interval (hours), so it fails the requirement.
A global (zone-sharded) cluster with members of every shard in an eu-west-1 zone: RPO collapses to seconds, no bespoke pipeline, and failover/reads are Atlas-managed. This is the recommendation.

Costs and consequences

Write latency: with w: "majority", cross-region acknowledgement adds roughly the us-east-1/eu-west-1 round trip to every write. Measure it; if unacceptable, downgrade only non-critical writes deliberately, not globally.
Shard key: the zone key must be part of the shard key prefix, forcing the key decision in the previous task.
Topology: you now need at least three nodes per shard region plus config servers, so the sharded tier cost is real, and it should be justified by the DR requirement as much as by load.

Prove it: quarterly DR drills - read from eu-west-1 with a zone-tagged read preference, kill a us-east-1 member, and record measured RPO/RTO rather than assumed ones.