Security, Resiliency, and Sustainability
This domain tests the operational qualities of an AI fabric: isolating tenants and management traffic, protecting data in flight, surviving component and congestion failures, and running GPU-dense infrastructure within a credible power and cooling envelope.
Security controls
- Segmentation: separate VRFs per tenant and per function (compute, storage, management, OOB), enforced with ACLs and, where deployed, Cisco TrustSec group tags so policy follows the workload rather than the subnet.
- Management-plane isolation: OOB reachable only through a jump host, AAA to TACACS+ or ISE, control-plane policing, and no data-plane path into OOB.
- Link encryption: MACsec on fabric links protects data in flight between switches and to the host where supported. It does not protect data at rest, and it does not replace tenant segmentation.
- AI-specific risk extends beyond the network: model artifacts, training data, and inference endpoints. Keep AI application security posture in scope alongside the underlay.
Resiliency
- Dual-homed servers via vPC, redundant super-spines, BFD for fast failure detection, and ISSU for hitless upgrades where supported.
- PFC watchdog and storm control are resilience features, not optional extras: a single malfunctioning NIC that keeps asserting pause can stall an entire lossless class.
- Training jobs should checkpoint frequently enough that a node failure costs minutes, not days. The storage design and the resiliency design must agree on the recovery point objective.
Sustainability
Rack power density drives both cost and feasibility. Practical levers: high-efficiency (Titanium) power supplies, hot-aisle containment, inlet temperatures held within ASHRAE recommended range rather than chilled to the minimum, GPU power capping during non-critical jobs, and per-rack power telemetry so idle capacity is visible. Refresh planning and hardware take-back programs address embodied carbon, which increasingly appears in procurement criteria.
Worked example: MACsec decision
A regulated tenant requires encryption in flight across the fabric. Options are MACsec hop-by-hop on 400G links or an overlay. MACsec on Nexus 9000 provides line-rate encryption with negligible latency and no overlay encapsulation overhead, but it requires consistent key management (MKA connectivity association) across every hop and platform support end to end. If one hop lacks support, the chain breaks and the requirement is not met - so verify per-platform support before committing to the design.
Traps that cost marks
- Deploying PFC without watchdog and blaming the fabric for a self-inflicted outage.
- Assuming MACsec covers data at rest.
- Leaving OOB reachable from the data plane.
- Planning cooling for nameplate power instead of measured GPU load.