Three Members, Two Availability Zones, One Tight Budget
A five-person platform team is deploying a new replica set for an order-management service. Budget allows three VMs: two in us-east-1a (8 vCPU, 64 GB RAM, local NVMe) and one in us-east-1b (2 vCPU, 16 GB RAM, EBS gp3). Peak load is 6,000 writes/sec of small updates and 20,000 reads/sec, with a p99 write target of 25 ms. The service currently runs writeConcern {w:1} and readPreference primary. Two proposals are on the table. Proposal A replaces the us-east-1b member with an arbiter on a small instance in us-east-1c, freeing budget for a larger primary. Proposal B keeps three data-bearing members, sets the third member's priority to 0, and points the analytics team's aggregation queries at it with readPreference secondaryPreferred. The team also wants to know whether moving to w:"majority" will blow the latency budget, given that the us-east-1b node fsyncs in 6 ms versus 0.2 ms on NVMe.
1. Evaluate Proposal A. Would you approve replacing the third data-bearing member with an arbiter?
Read the worked answer
No. Work through the vote arithmetic first. With three voting members, majority is two. After the swap the set is two data-bearing members plus one arbiter, so primary + arbiter still form a majority even when one data node is down. The set will therefore keep accepting writes with exactly one copy of the data on disk. If that remaining data node then fails before the downed member returns, everything acknowledged in that window is gone and recovery produces rollback files.
The zone layout is worse than the vote math. Two data-bearing members in us-east-1a plus an arbiter in us-east-1c is a single-availability-zone data footprint: losing us-east-1a is a full outage with no second copy anywhere. An arbiter also cannot be hidden, delayed, or given a priority, holds no data for analytics reads, and adds no read capacity.
There is an operational cost too: going back to three data-bearing members later means removing the arbiter and running an initial sync on a new member, which is I/O-heavy and risks cache eviction on the donor.
Recommend instead: three data-bearing members spread across three AZs, with the smaller instance used as a low-priority data member rather than an arbiter. If cost truly forbids three large hosts, shrink all three evenly and bound latency with wtimeout rather than trading durability for capacity.
2. Evaluate Proposal B. What would you change about sending analytics aggregation reads to the third member?
Read the worked answer
The intent is sound but the mechanism is too loose. secondaryPreferred falls back to the primary whenever no secondary is reachable, so a failover or a lagging member silently dumps 20,000 reads/sec of aggregations onto the node that is serving writes. Set the analytics target to priority 0 and hidden:true so it can never be elected, and address it with explicit tags, for example readPreference {mode:"secondary", tags:[{node:"analytics"}]}. Unlike secondaryPreferred, a tagged read that finds no matching member fails fast instead of quietly hitting the primary.
Second, size it for the load. Aggregations that scan large ranges will churn the WiredTiger cache on that member, and cache pressure on a secondary translates directly into replication lag, because oplog application competes for the same cache and I/O. A 2 vCPU / 16 GB EBS host is unlikely to absorb 20,000 reads/sec plus 6,000 writes/sec of replication; it will fall behind during peaks and periodically need a full resync.
Third, cap staleness with maxStalenessSeconds so a lagging analytics node stops serving rather than returning very old data, and alert on rs.printSecondaryReplicationInfo() and the member's oplog apply rate.
If analytics is genuinely a large workload, the durable answer is a separate replica set fed by change streams, not a shared member of the operational set.
3. The team moves to writeConcern {w:"majority", j:true}. Will the 6 ms EBS fsync on the third member dominate p99 write latency?
Read the worked answer
Not in the normal case, and this is the key insight. With three voting members, w:"majority" waits until a majority of members have acknowledged (two of three with j:true), and that majority is satisfied by the primary plus whichever secondary journals first. It is not an average over all members and it is not the slowest member. In steady state the primary and the local NVMe secondary in us-east-1a will satisfy majority in well under a millisecond of journal latency, so the 6 ms EBS member is not on the critical path.
It becomes the critical path exactly when you can least afford it: when the fast secondary is down, in catch-up, or partitioned. Then majority requires the EBS member, and p99 jumps to roughly its fsync time plus cross-AZ round trip. That is still far better than w:1, but it should be measured, not assumed.
Practical plan: deploy with {w:"majority", j:true, wtimeout:500}, confirm writeConcernMajorityJournalDefault is true, and watch the delta between w:1 and w:majority latency under load and again with one secondary stopped. Also confirm retryWrites is enabled in the driver so elections surface as retried operations rather than application errors. If the degraded-mode tail is unacceptable, either keep three similarly fast data members or accept a bounded w:2 with j:true and document the trade-off.