Second Node Fails to Join After Back-End CRC Errors
A technician is installing a four-node PowerScale cluster in a remote data center. The first node is configured with OneFS 9.7; back-end interfaces are int-a 10.10.0.11/24 and int-b 10.10.1.11/24. The two 100 GbE back-end switches use VLAN 110, MTU 9216, and no LACP. The second node is addressed as 10.10.0.12 and 10.10.1.12. After the join command, the second node reboots but never appears in isi status; the first node shows LNN 2 as down. Switch logs show CRC errors on port 17, which connects to the second node. The technician has already replaced the cable and SFP. To save time, they propose changing the back-end MTU to 1500 to match the front-end and disabling flow control on all back-end ports. Remote hands are available for only two more hours. The cluster is 40 km from the main site.
1. What are the most likely causes of the join failure, and what should be checked next before changing cluster-wide settings?
Read the worked answer
CRC errors point to the physical layer or switch port configuration, not to a OneFS join bug. Since the cable and SFP were replaced, check the switch port itself, optics compatibility, transceiver type, bend radius, and whether the port is configured correctly for 100 GbE. Verify that the second node's back-end interfaces are actually up, that the IP addresses are unique, and that both nodes use the same VLAN and MTU. Use the serial console or console log to confirm the node boots and attempts to join. Check switch logs for link flaps, FEC mismatches, or incorrect breakout settings. Do not change MTU or flow control until the physical link is clean, because those changes affect the whole back-end network.
2. Evaluate the proposal to set back-end MTU to 1500 and disable flow control. What is the correct approach?
Read the worked answer
Lowering the back-end MTU to 1500 is not a safe shortcut. OneFS back-end networks require consistent jumbo frames, typically MTU 9216, on every node port and switch port. A mismatch can cause join failures, cluster instability, or intermittent node-down events. Disabling flow control on 100 GbE back-end ports is also risky because it can increase drops during bursts. The correct approach is to keep the intended MTU and fix the port causing CRC errors. Validate MTU with a large-payload ping with the do-not-fragment bit, confirm the switch port MTU, and only then rejoin the node.
3. Remote hands have two hours. What sequence would you follow to restore the second node?
Read the worked answer
First, have remote hands move the second node to a known-good switch port and replace the optic on the switch side, not only the node side. Second, confirm link state, VLAN 110, MTU 9216, and FEC settings match the first node's ports. Third, use the serial console to confirm the node boots and its int-a and int-b addresses are correct. Fourth, re-run the join from the first node and watch isi status and switch logs. If the node still fails, capture logs before the remote hands leave. Do not change cluster-wide MTU or flow control during the window; make one controlled change at a time so the cause is provable.