Priority 10, Eight-Millisecond Disks, and a Stale Delayed Member
A five-member replica set (three voting data-bearing members, one priority-0 member in a second region, one hidden delayed member with slaveDelay 3600) serves a ticketing system. Last month an operator set priority: 10 on node B, a 32 GB host with network-attached storage whose fsync latency is 8 ms, versus 0.1 ms on the NVMe hosts. After B was patched and restarted, it won the next election and stayed primary. Application p99 write latency went from 18 ms to 340 ms and has stayed there; B's CPU is only 25%. Meanwhile node D, the delayed member, reports stateStr RECOVERING with 'too stale to catch up' after a three-day outage. The team also reports four unscheduled elections in the past week and wants to know whether that rate is acceptable.
1. Explain why node B keeps winning elections, why latency rose despite low CPU, and how to correct it with minimal disruption.
Read the worked answer
Elections are won on priority, not on hardware quality. With priority 10 against a field of 1s, B is preferred by every voting member, and it stays preferred as long as it is up and current. Restarting it made it catch up and then immediately trigger an election it could not lose. Low CPU is expected: the bottleneck is fsync and journal commit latency on the network storage, not compute.
The latency consequence is structural. Every write that reaches w:"majority" with j:true waits for the primary's own journal commit before it can even acknowledge, and then for replication. On a primary that fsyncs in 8 ms, the floor for each write is roughly that fsync, so a few hundred writes per connection queue into hundreds of milliseconds of p99. The NVMe members were fine secondaries but are now irrelevant to the write path.
Correction, in order, without a maintenance window:
- Reconfig priorities so B is 0 or 1 and an NVMe member is highest: build the config document, call rs.reconfig(cfg), verify with rs.conf(). A reconfig does not by itself force a stepdown.
- Then call rs.stepDown(60, 30) on B so a preferred member takes over cleanly; the secondaryCatchUpPeriodSecs value keeps applications from stalling.
- Keep B as a data-bearing member with priority 0 if it still adds a geography, or replace the storage.
Only use rs.reconfig(cfg, {force:true}) when a majority of voters is unreachable, and expect rollback risk when you do.
2. Node D has been down for three days and now cannot catch up. What is your recovery path, and how do you choose the sync source?
Read the worked answer
First confirm the diagnosis rather than guessing: compare the oplog window (rs.printReplicationInfo() on the primary) with D's last applied optime. If the oldest oplog entry on every member is newer than D's last optime, no amount of waiting will help, and oplog application will never close the gap. That is the 'too stale' condition.
Recovery options, cheapest first:
- If any member still holds an oplog entry older than D's last applied optime, point D at that member explicitly with rs.syncFrom("host:port") and restart. This is rare after three days.
- Otherwise D must do a full initial sync: stop mongod, move or remove the contents of dbPath, restart, and let it copy every collection and build indexes from a donor.
Choose the donor deliberately. Do not let D sync from the primary by default; on a busy set that steals cache and I/O from the writer. Use rs.syncFrom() to target a healthy secondary, ideally one on the same network segment, and prefer a member that is not itself lagging. Watch disk headroom on the donor: initial sync reads everything.
Expect the sync to take hours for a large data set, and expect it to fail if it falls too far behind during the copy. While it runs, keep D out of the voting picture by leaving it priority 0, and alert on its replication lag afterwards.
Because D is the delayed member, also verify the delay setting survived, and document that a delayed member that is offline longer than the oplog window has to be reseeded like any other.
3. Four unscheduled elections in a week - what is the baseline, how do you diagnose the cause, and what would you change?
Read the worked answer
The baseline for a stable replica set is zero unscheduled elections. Elections are disruptive: the primary closes connections, in-flight writes that were not majority-acknowledged can roll back, and drivers must rediscover topology. Four in a week is a symptom, not a configuration to tune around.
Diagnose from three sources:
- rs.status() for each member's state, lastHeartbeatMessage, and the electionDate fields - this tells you which member called and who supported it.
- The mongod log around each timestamp, looking for 'Starting an election', 'Not starting an election', 'replSet info starting election', and the pre-election lines about heartbeat timeouts or priority takeovers.
- serverStatus().elections.called and the repl metrics, plus host-level evidence: CPU steal, disk stalls over 2 s, packet loss or MTU problems between AZs.
The usual culprits are a primary that misses heartbeats because its storage stalls (checkpoint or fsync spikes), a member with a high priority that flaps, network partitions between availability zones, and administrators running stepDown or reconfig without knowing it. With heartbeatIntervalMillis at 2 s and electionTimeoutMillis at 10 s, a storage stall of a few seconds is enough to lose an election.
Fixes follow the cause: correct priorities so one preferred member exists and the rest are 0 or 1; move data off failing storage; fix the network path; and standardize maintenance so stepDown happens at a known time. Then confirm you have gone a month with elections.called unchanged before declaring it resolved.