Two Peers Down: Search Coverage Collapses While Data Survives
A single-site indexer cluster runs six peers behind one cluster master, with replication_factor = 3 and search_factor = 2, holding roughly 900 buckets. On Friday night a PDU failure takes idx-04 and idx-05 offline simultaneously; both are still down on Monday morning. The master shows several hundred buckets in a not-searchable state, and the security team reports that dashboards over the last ten days return partial results while some older ranges look complete. The infrastructure lead proposes setting search_factor = 1 'to get search back immediately' and replication_factor = 2 'to speed up recovery.' The remaining four peers are healthy and disk utilization is 61%.
1. Explain why the affected buckets are no longer searchable, state whether any data has been lost, and identify the correct remediation.
Read the worked answer
The two factors control different things.
- replication_factor = 3 means three raw copies of every bucket exist across the peer set. Losing two peers out of six should not destroy data, because each bucket has one surviving copy at worst.
- search_factor = 2 means two of those copies must be searchable copies (copies the search head can read directly).
A bucket is searchable only when it meets the search factor. With two peers down, any bucket whose three copies were distributed so that two landed on idx-04 and idx-05 is left with a single copy. One copy < search factor 2, so the master marks the bucket not-searchable and search heads skip it - that is why results are partial and why only certain time ranges (the ranges served by those buckets) are affected.
No data is lost yet. The surviving copy is intact; the cluster is at risk only if a third peer holding that copy fails.
Correct remediation: restore or replace idx-04 and idx-05. When a peer rejoins, the master automatically runs fixup - re-replicating raw copies to satisfy RF and promoting copies to satisfy SF. Monitor with splunk show cluster-status on the master (look for fixup tasks in progress) and splunk show cluster-bundle-status to confirm no pending config work. Do not attempt to 'repair' buckets manually.
2. Evaluate the proposal to set search_factor = 1 and replication_factor = 2. What would actually happen, and what is the danger?
Read the worked answer
search_factor = 1 would technically restore search for the one-copy buckets, because a single copy would satisfy the search factor. That is the only argument in its favor. The cost is severe: with SF=1 there is no search redundancy at all, so the next single peer failure makes whole bucket ranges unsearchable, and there is no healthy alternative copy for the search head to fall back to.
replication_factor = 2 with SF=2 is worse. The cluster requires RF ≥ SF, and at RF = SF = 2 every copy must be searchable - a single peer loss removes half of the copies and immediately breaks searchability. It tolerates zero peer failures. It also cannot even be applied cleanly right now: only four peers are live, so the cluster would start a large, unnecessary re-replication cycle across already-degraded hardware, competing with fixup and with search I/O.
Both changes are also the wrong instinct because they treat a hardware event as a configuration problem. Reducing factors lowers the cluster's fault tolerance permanently to work around a transient outage; when the peers return, you must raise them again, triggering more replication churn.
Recommended: keep RF=3 / SF=2, restore the two peers, and for future planned work use maintenance mode on peers rather than letting them drop out abruptly.
3. Before the team touches the four remaining peers for maintenance, what should they do, and how do they confirm the cluster is healthy again afterwards?
Read the worked answer
Use maintenance mode on each peer being serviced (from the cluster master UI, or splunk enable maintenance-mode on the peer) rather than simply stopping splunkd. Maintenance mode tells the master to suppress fixup and re-replication for that peer, so the cluster does not thrash - repeatedly marking buckets non-searchable and re-replicating - every time a peer briefly disappears.
Sequence:
- Wait until the master reports a fully healthy cluster from the PDU event first; never stack a second disruption on an unfinished fixup.
- Put one peer at a time into maintenance mode (not all four).
- Do the work, bring the peer back, take it out of maintenance mode, and confirm fixup has completed before moving to the next peer.
Verification on the master:
splunk show cluster-status- all peers Up, no buckets in the 'not searchable' or 'replication needed' counts, fixup tasks at 0.splunk show cluster-bundle-status- last bundle applied successfully on every peer with no restart-required state.- Spot-check from a search head that a known time range returns complete results (event counts per host should match the source systems).
This matters because maintenance mode is the difference between a controlled single-peer outage and an uncontrolled cluster-wide fixup storm.