MTU 9000 Rollout Script Fails at Host 73

Failure modes, in order of impact:

1. set -e at fleet scope. One host's failure stops the rollout, producing a half-changed fleet with no record of which hosts were changed. There is no trap and no summary.
2. No per-host timeout. ssh without ConnectTimeout and server-alive settings can hang for minutes or hours; one unresponsive host stops everything.
3. Remote && chain hides partial success. If ip link set succeeds and systemctl restart nfs-client fails, the host is left with MTU 9000 and a broken mount, and the script only sees a nonzero exit.
4. Word splitting and CRLF. for h in $(cat hosts.txt) splits on whitespace, performs globbing, and keeps \r. The carriage-return host fails DNS or SSH with a confusing error. Unquoted root@$h is also an injection risk if the file contains options.
5. No preflight. Jumbo frames require every hop to support 9000. Changing the host before verifying end-to-end guarantees the array will drop the larger frames and NFS will hang.
6. Non-idempotent. A second run re-applies the MTU and restarts NFS on hosts that are already correct, causing avoidable outages, and still dies on the same host key mismatch.
7. Host key policy. BatchMode=yes does not disable StrictHostKeyChecking; the mismatch is expected behavior and should be a handled exception, not a global abort.

set -e plus a remote && chain is a poor fit because the unit of success is a host, not an individual command. You need per-host status capture, continue-on-error, and an explicit summary.

Separate the per-host action from the orchestration and record status per host:

#!/usr/bin/env bash
set -uo pipefail
inventory=${1:?usage: rollout.sh inventory.txt}
log=mtu-rollout-$(date +%Y%m%dT%H%M%S).log
: >"$log"

check_and_set() {
  local host=${1%$'\r'} out rc
  [[ -z $host || $host == \#* ]] && return 0
  out=$(timeout 45 ssh -o BatchMode=yes -o ConnectTimeout=8 \
        -o ServerAliveInterval=5 -o ServerAliveCountMax=2 \
        -o StrictHostKeyChecking=accept-new -- "$host" '
      set -e
      cur=$(cat /sys/class/net/ens5/mtu)
      if [ "$cur" -eq 9000 ]; then echo already-9000; exit 0; fi
      ping -M do -c 1 -W 2 -s 8972 198.51.100.20 >/dev/null || \
        { echo jumbo-preflight-failed; exit 20; }
      ip link set ens5 mtu 9000
      systemctl restart nfs-client
      echo mtu-ok
    ' 2>&1); rc=$?
  printf '%s\t%s\t%s\n' "$host" "$rc" "$(tr '\n' ' ' <<<"$out")" | tee -a "$log"
  return 0
}
export -f check_and_set; export log
tr -d '\r' <"$inventory" | xargs -r -P 8 -n 1 bash -c 'check_and_set "$@"' _

Key points:

• set -uo pipefail without -e: failures are recorded, not fatal to the run.
• timeout bounds each host, and the SSH options bound the connection and dead-peer detection.
• The remote script is idempotent and preflights jumbo reachability to the array before changing the MTU, so a switch port still at 1500 fails safely with a distinct exit code.
• ip link set is runtime only; persist it with NetworkManager (nmcli con mod ... 802-3-ethernet.mtu 9000) or a systemd-networkd .network file, otherwise it reverts on reboot.
• For a true fleet preflight, first run a read-only pass that collects ip -o link show ens5, ip route get 198.51.100.20, and switch port MTU from inventory or LLDP data.

Make the second run state-driven, not hope-driven.

1. Reconcile the inventory first: strip CR, drop comments and blanks, and validate each host with getent hosts and a TCP check. Remove the two decommissioned hosts so they do not consume retries.
2. Build a state file from the first log. Classify each host as unchanged, changed and healthy, or changed and broken. Distinct remote exit codes (20 = jumbo preflight failed, 21 = NFS restart failed) make classification deterministic.
3. For changed-but-broken hosts, roll back first: ip link set ens5 mtu 1500 and systemctl restart nfs-client, then verify with mountpoint -q /mnt/storage and stat /mnt/storage. Rollback does not require jumbo to work, so it is safe even when NFS is hung. Retry the change only after the mount is healthy.
4. Run a dry-run pass with the same code but no ip link set. It should report already-9000 for successful hosts and the exact preflight failure for the rest. That is your idempotency check.
5. Apply in waves of 10-20 with a pause and a storage-array health check between waves. Watch switch counters for giants, CRC errors, and drops; a port left at 1500 shows up on the array side.
6. Fix host 73's host key before retrying: verify the new key out of band, then ssh-keygen -R <host> and re-add. After the bootstrap, move to StrictHostKeyChecking=yes with a managed known_hosts.
7. Keep the before/after MTU and the per-host log as the audit trail. The script should exit nonzero only if the final summary contains failures, not on the first one.