A rebalance that stalls while a disk is dying

What a rebalance actually does: it moves vBuckets between data nodes (1024 per bucket), and for the index service it must re-stream and rebuild index data on whichever node now owns each index. Index builds are CPU- and disk-intensive and compete with the same NVMe devices that serve queries.

Why this one is slow:

• idx_orders_status is 240 GB with zero index replicas, so it lives on exactly one node and must be rebuilt when placement changes.
• All services share the same six nodes, so index build I/O starves the data service and the query service simultaneously.
• 900 GB with one replica means roughly 1.8 GB of vBucket data moving, with 2,800 ops/s of live traffic throughout.
• Reaching only 62% after nine hours points at index rebuilds, not vBucket movement, as the tail.

Why p99 went from 40 ms to 4.5 s: the disks are the bottleneck. Queries that normally hit cache or short queues now wait behind index build writes, and the index nodes look worst because that is where the work is. Nothing is logically broken - the cluster is I/O-saturated, and adding load will deepen it.

Do not add index replicas now: creating them triggers additional builds on already saturated disks and will extend the rebalance. Change topology after this operation ends, not during it.

The failing disk is the priority. A 96%-full disk with SMART errors is a node that may vanish mid-rebalance, and with one replica an unplanned loss is an availability event, not merely a slowdown.

Constraints to respect:

• A rebalance cannot be paused. The choices are let it run or abort it.
• Failover is not permitted while a rebalance is in progress, and auto-failover is suppressed during rebalance, so you cannot fail the node out from underneath it.
• Aborting is safe: vBuckets already moved stay moved, the cluster remains consistent, and you lose the progress but not the data.

Recommendation: abort the rebalance now. You have a hardware fault and a rebalance that will not finish before the disk gives out. Then:

1. Confirm every vBucket hosted on the affected node has an in-sync replica.
2. If the node still responds, perform a graceful failover so its vBuckets are handed over cleanly.
3. Replace the node in a swap rebalance rather than adding nodes and rebalancing separately.

If the disk dies first: the node drops out, auto-failover fires after 120 s, replicas are promoted, and you run with reduced redundancy and elevated latency while indexes on that node rebuild. Survivable at replica 1, but with no margin - which is exactly why the abort decision should be made now rather than observed later.

In order:

1. Add index replicas (=1) on the large indexes before the next rebalance, in a quiet window. This costs disk (roughly doubling index footprint - budget about 240 GB more for the big index) and doubles build work, which is precisely why it must not happen during a rebalance. The payoff is index availability through failover and rebalance: with a replica, an index no longer has to be rebuilt from scratch when its node leaves.
2. Dedicate index nodes. Move the index service onto two or three nodes that run nothing else, so builds stop competing with the data service for I/O.
3. Consider partitioned indexes for the 240 GB index so a build parallelizes across index nodes instead of rebuilding one monolith. The partition key must be a field the query filters on, or queries scatter-gather across partitions.
4. Use swap rebalance: new nodes join and old nodes leave in a single operation, and data moves once instead of twice.
5. Sequence and guard: fix capacity headroom first, then index replicas, then the swap rebalance in a maintenance window, with query timeouts and a circuit breaker at the application edge.

Tradeoff to state explicitly: index replicas improve availability, not query throughput - reads still go to the active partition and the standby exists for failover and rebalance. Partitioned indexes are what improve build time and parallel scan. Adding replicas without fixing service placement just moves the contention.