Triage After a Disk Failure and Network Change
An eight-node Cohesity cluster has 1.6 PB raw capacity, replication factor 2, and is 68% full. One node reports a failed disk and begins rebuilding. After the rebuild starts, VMware backup throughput falls by 40%, several backup jobs fail with timeout errors, and the Cohesity UI shows intermittent authentication failures for Active Directory-integrated users. Two nodes have an NTP offset of seven minutes. A new top-of-rack switch was installed last week, and one node's interface is configured with an MTU of 9000 while the others use 1500. The team wants to add a ninth node, enable erasure coding, and open a support case. The failed disk is still in the cluster, and the rebuild is 18% complete. Backup success rate for the last 24 hours is 82%, down from 99%.
1. What is your triage order, and which actions should wait? Justify by risk to data durability versus performance.
Read the worked answer
Triage by risk to data durability first, then authentication and control-plane stability, then throughput. Performance is important, but a cluster that loses redundancy or loses administrative access is a bigger risk.
- Confirm data durability: check cluster health, the failed disk state, rebuild progress, and whether any other disks or nodes are degraded. Verify that replication factor 2 still has a complete copy of every extent. Do not remove the failed disk or start intrusive maintenance while the rebuild is only 18% complete.
- Stabilize authentication: the seven-minute NTP offset can cause Kerberos and AD authentication failures. Correct time synchronization on the two nodes and verify that the cluster services recover. This is a control-plane issue and may also affect backups that use AD credentials.
- Check the network change: the MTU mismatch is a likely cause of timeouts and throughput loss. Confirm end-to-end MTU on the new switch and all node interfaces. A single 9000-MTU interface among 1500-MTU peers can cause fragmentation, dropped packets, and retransmits.
- Investigate throughput: only after durability and authentication are stable, determine how much of the 40% drop is rebuild I/O and how much is network retransmission.
Actions that should wait: adding a ninth node and enabling erasure coding. Both change data placement and add I/O during an active rebuild. Open a support case early, but do not make multiple structural changes at once.
2. What are the likely causes of the throughput drop and authentication failures, and how would you confirm each?
Read the worked answer
The throughput drop and authentication failures probably have different causes, and treating them as one problem will slow diagnosis.
Authentication failures: a seven-minute NTP offset is outside the default tolerance for Kerberos, so AD-integrated users and services can fail intermittently. Confirm by comparing time on the affected nodes, checking the cluster's time-source configuration, and reviewing authentication logs around the failures. Correcting NTP and restarting the relevant services should reduce the failures; if it does not, check DNS and the service account used for AD integration.
Throughput drop: two likely contributors are the rebuild and the MTU mismatch. A disk rebuild consumes I/O and network bandwidth, so a 40% drop during rebuild is plausible but should be measured, not assumed. The new switch and the 9000-versus-1500 MTU mismatch can cause fragmentation and retransmits, especially for large backup streams. Confirm with interface error counters, drop counters, MTU tests between nodes, and switch logs. Also check whether the backup failures correlate with specific nodes or subnets.
Timeout errors: distinguish storage timeouts from network timeouts. Review job logs for the failing phase: snapshot, data transfer, or metadata update. If failures follow the node with the MTU mismatch, the network is the stronger hypothesis.
Change one variable at a time. Correct NTP, verify MTU, and observe whether throughput recovers before adding capacity.
3. Should the team add a node and enable erasure coding now? Explain capacity and rebuild implications.
Read the worked answer
No. Adding a node and enabling erasure coding during an active rebuild are both disruptive changes that alter data placement and increase I/O. The cluster is already at 68% used with replication factor 2, and the rebuild is only 18% complete. The first priority is to restore redundancy and stability.
Capacity: 68% used on 1.6 PB raw with RF2 leaves limited headroom. Rebuilds consume space and I/O, and a new node triggers rebalancing. Adding a node may be appropriate soon if capacity or performance is projected to remain constrained, but it should be planned after the rebuild completes and after the network issue is resolved. Otherwise the team will not know whether the new node improved anything or simply added more background work.
Erasure coding: enabling erasure coding changes the protection scheme and requires data movement. Doing it during a rebuild can lengthen the rebuild, increase risk, and make performance troubleshooting ambiguous. Wait until the cluster is healthy, then evaluate erasure coding against the capacity savings and performance impact for the workload.
Support case: open it now with a support bundle, timeline, disk failure details, NTP offset, MTU mismatch, and job failure samples. Support can confirm whether the rebuild behavior is expected. The team should avoid speculative structural changes before support reviews the logs. The correct sequence is stabilize, verify, then optimize capacity.