From Spreadsheet to GitOps: Onboarding 40 AI Tenants Across Three Data Centers
An enterprise has 96 Nexus switches across three data centers, 40 internal AI tenants, and a six-week lead time for new tenant capacity. Today engineers configure VLANs, VRF leaking, QoS, and ACLs by CLI from a spreadsheet; a recent audit found 17% drift between intended and running configuration, including a tenant ACL that exposed a storage subnet. The company wants VXLAN EVPN for tenant isolation, automated Day-0 and Day-1 provisioning, and a repeatable way to prove compliance. It has Cisco Nexus Dashboard, Intersight, Ansible, and a Git repository, but no source of truth and no CI pipeline. Security requires TACACS+ command authorization, per-tenant RBAC, and encrypted management traffic. The CIO wants new tenants live in three days, not six weeks.
1. Design the automation architecture. What owns intent, what renders configuration, and what touches the devices?
Read the worked answer
Separate intent, rendering, and execution. The source of truth should be a structured inventory, for example NetBox or a Git-backed YAML model, containing sites, fabrics, switches, tenants, VRFs, VLANs, subnets, anycast gateways, QoS classes, and ACL policy. Git is the audit trail and the review mechanism; the source of truth is the data model that the pipeline reads, not a spreadsheet.
Render configuration with Ansible or Terraform templates, but do not let templates improvise. For VXLAN EVPN, Nexus Dashboard Fabric Controller should own fabric underlay, overlay, and tenant provisioning so that the fabric state machine handles BGP, VNI, and anycast gateway consistency. For compute, Intersight should own server profiles and policies. The pipeline's job is to validate the model, call the controller APIs, and verify the result, not to push raw CLI.
Day-0 uses PnP or ZTP with signed images and a golden configuration. Day-1 uses NDFC tenant and network templates. Day-2 uses compliance and telemetry. Multi-site consistency should use Nexus Dashboard Orchestrator so a tenant template is instantiated per site with local differences explicit. The pipeline stages should be: schema validation, policy lint, dry-run or diff, canary site, full deploy, post-check, and rollback. No stage should be skippable for a production tenant.
2. How do you meet the security requirements without making the pipeline the weakest link?
Read the worked answer
Treat the pipeline as a privileged system. Use TACACS+ with command authorization on every device and per-tenant RBAC so a tenant operator cannot push a fabric-wide change. Separate roles: fabric admin, tenant admin, and auditor. The automation account should have only the permissions required for its templates, not full admin, and credentials should come from a vault at runtime, never from Git.
Encrypt management traffic with SSH and HTTPS, disable Telnet and HTTP, and restrict management access with control-plane ACLs. For fabric links, evaluate MACsec where the platform supports it; not every 400G optic or line card does, so verify the hardware before promising it. Sign Git commits and require review for tenant policy changes. The pipeline should produce an artifact, such as a rendered intent bundle and a diff, that is approved before execution.
The storage-subnet exposure must be handled as a policy bug, not a one-off fix. Encode tenant isolation as a test: a tenant ACL must not permit the storage prefix, and the pipeline should fail closed if the test fails. Add a post-deploy verification that reads the running ACL and compares it to intent. Compliance reports should be generated automatically, because a manual audit found the drift only after the fact.
3. The CIO wants three-day tenant turn-up. What operating model and metrics would make that credible, and what would you refuse to automate?
Read the worked answer
Three days is credible only if most of the work moves before the request. Pre-build tenant templates for common profiles, pre-provision fabric capacity and VNI ranges, and make the request a pull request against the data model. Day one is validation and approval, day two is canary deployment and verification, day three is production rollout and handover. The six-week lead time is mostly procurement and manual coordination; automation fixes the configuration half, not the hardware half, so capacity planning must be separated from tenant onboarding.
Metrics should be lead time from approved request to verified tenant, percentage of changes deployed through the pipeline, drift count, mean time to detect drift, and failed-change rollback rate. Track policy-test pass rate and the number of emergency CLI changes; emergency changes are the leading indicator of a broken model.
Refuse to automate destructive actions without a human approval gate: fabric-wide underlay changes, VNI range reallocation, ACLs that affect shared storage, and any change that touches more than one site at once. Also refuse to let the pipeline auto-remediate drift until the detection has a low false-positive rate. Start with report-only, then canary remediation, then closed-loop for a narrow set of low-risk policies. Do not automate around a missing source of truth; that just makes drift faster.