AI Factory Deployment and Management
AI Factory Deployment and Management
This domain covers turning bare metal into a validated, multi-tenant AI factory: reference architecture, provisioning, firmware baselines, scheduling, and acceptance testing. Base Command Manager, Kubernetes device plugins, and schedulers such as Run:ai are common context.
Core rules and concepts
- Provision from a golden image and pin firmware, driver, and CUDA versions across the cluster; drift breaks reproducibility.
- Separate compute, storage, in-band management, and out-of-band networks.
- Validate in layers: node burn-in, fabric collectives, then a representative workload.
- Use Kubernetes device plugins with a scheduler for quotas, gang scheduling, and MIG allocation.
- Change safely: cordon and drain nodes, canary firmware, and keep a rollback image. Record the validated baseline for the next cluster.
Worked example
A 32-GPU cluster has four nodes, each with eight H100 GPUs. Acceptance runs NCCL all-reduce across all 32 GPUs, not just per-node tests. If one node reports roughly half the bus bandwidth of its peers, inspect NVLink/NVSwitch topology and rail NIC mapping before sign-off. The reasoning: one slow node limits every collective job, so acceptance must exercise the full fabric path.
Traps that cost marks
Skipping burn-in; mixing firmware across nodes; putting storage traffic on the management VLAN; assuming a container image ships the host driver; and validating only a single-GPU benchmark.