UCS Fabric Failover and Uplink Oversubscription in a 32-Blade Cluster
A company runs 32 Cisco UCS B200 M6 blades in four 5108 chassis, managed by two Fabric Interconnects in cluster mode. Each blade has a VIC 1440 with two 25-GbE ports to the FIs. Each FI has four 40-GbE uplinks to a pair of Nexus 93180YC-FX switches. ESXi hosts use two vNICs per service profile; fabric failover is disabled and vNIC-A is pinned to Fabric A, vNIC-B to Fabric B. During a recent FI firmware upgrade, nine ESXi hosts lost all storage paths and vMotion failed because only one vNIC remained active. The team also plans to add eight more blades and wants to know whether the 40-GbE uplinks are adequate. Current peak egress is 110 Gbps per FI. Service profiles are built from a template with one vNIC template and one vHBA template.
1. Explain why the fabric-failover and pinning choices contributed to the outage, and recommend a safer vNIC design for ESXi.
Read the worked answer
With pinning, the host owns failover. If vNIC-A was the only active uplink in the vSwitch and vNIC-B was standby but not configured to take over, an FI upgrade that resets Fabric A leaves the host with no working path even though a second physical adapter exists. Fabric failover moves that responsibility to the FI: the vNIC stays associated with one fabric, but the FI can forward traffic on the other fabric if the primary path fails, and the host still sees link up. For ESXi, the cleaner design is usually active/active vNICs across the two fabrics with an upstream vPC domain, or fabric failover only after validating VMware and upstream switch support. Check the vSwitch teaming policy, the service profile vNIC placement, and the upstream port-channel configuration. Do not simply enable fabric failover on every vNIC as a blanket fix; it can hide path failures and is not supported with every teaming mode. The immediate corrective action is to make both vNICs usable and test an FI upgrade in a maintenance window.
2. Are the four 40-GbE uplinks per FI adequate after eight more blades are added? Show the calculation and recommend a target.
Read the worked answer
Current host-side capacity per FI is 32 blades times 25 Gb, or 800 Gb. Four 40-GbE uplinks provide 160 Gb, so the oversubscription is 800 divided by 160, or 5:1. Current peak egress of 110 Gb is about 69% of 160 Gb; that leaves limited burst headroom for vMotion and storage. After eight more blades, host-side capacity becomes 40 times 25 Gb, or 1000 Gb, and 1000 divided by 160 is 6.25:1. For a VMware cluster with shared storage, that is too thin. Adding two 100-GbE uplinks per FI raises uplink capacity to 360 Gb and brings oversubscription to about 2.8:1; four 100-GbE uplinks would give 560 Gb and about 1.8:1. Use LACP port channels, confirm the Nexus side has sufficient buffer and QoS, and monitor incast during vMotion and backup windows.
3. How should the service-profile template be changed so that vNIC and vHBA placement is consistent and repeatable?
Read the worked answer
Build separate vNIC templates for Fabric A and Fabric B, each with the correct fabric ID, VLAN, MTU, QoS policy, and failover setting. Build separate vHBA templates pinned to Fabric A and Fabric B with distinct WWPN pools and VSANs. Attach these to a service-profile template that also includes boot policy, BIOS policy, scrub policy, LAN connectivity policy, and SAN connectivity policy. Derive service profiles from that template rather than editing each blade by hand. Before re-associating, verify that the WWPN pools have enough addresses, that the vNIC failover behavior matches the ESXi teaming policy, and that the upstream vPC has both fabrics reachable. This prevents the situation where a single vNIC template leaves one path inactive or misconfigured.