LCM Upgrade Blocked by No-Egress Security Policy
Aster Logistics runs a six-node NC2 cluster in us-west-2 with 420 VMs, AOS 6.7, and Prism Central 2024.1. Storage is 70% full; memory pressure alerts fire at 85% every afternoon. The security team removed the NAT gateway and allows no internet egress from the NC2 VPC. LCM inventory now fails with a download.nutanix.com timeout, and a CVE fix requires upgrading AOS and Prism Central within 30 days. NCC also reports CVM time skew of 8 minutes and a pending firmware update. The team has an internal artifact server but has never used it for LCM. They can add one node, but finance wants evidence before approving it. They ask for a safe upgrade path that does not require opening internet access and does not take the cluster offline.
1. Plan the LCM configuration and upgrade sequence for the no-egress environment.
Read the worked answer
Do not open internet egress. Use Nutanix LCM dark-site capability: populate the internal artifact server with the required AOS, Prism Central, and firmware binaries and metadata, then configure LCM in Prism Central to use that internal repository. Validate connectivity from Prism Central and the CVMs to the artifact server, and confirm the repository signature or checksum so LCM does not reject the binaries. Sequence matters: run NCC checks, back up Prism Central, upgrade Prism Central to a version compatible with the target AOS, then upgrade AOS through LCM one node at a time so the cluster stays online. Confirm the target AOS and Prism Central compatibility before starting, and verify firmware compatibility in the same matrix. Schedule the work outside the afternoon memory-pressure window and keep a rollback plan. If the artifact server is incomplete, stop before starting rather than leaving a partially upgraded cluster.
2. How should the team address the time skew and memory pressure before the upgrade, and what evidence should finance see for the extra node?
Read the worked answer
Fix time skew first because Kerberos, AD integration, and LCM can fail when CVMs are more than a few minutes apart. Point the CVMs and Prism Central at internal NTP servers, verify chrony status, and confirm the skew is gone before any upgrade. For memory pressure, identify whether it is caused by overprovisioned VMs, a few hot workloads, or real aggregate demand. Pull at least a week of Prism Central metrics showing sustained memory use above 80–85% during business hours, plus the projected headroom needed for an N+1 event during a rolling upgrade. That evidence justifies the seventh node. Adding the node before the upgrade also gives the cluster room to evacuate a node safely. If finance declines, reduce memory pressure by right-sizing or moving batch workloads, and document the reduced resilience.
3. What monitoring and post-upgrade checks should be in place for this cluster?
Read the worked answer
Monitor at three layers. Nutanix NCC and Prism Central alerts should cover CVM health, disk capacity, memory pressure, LCM task status, and time skew. AWS CloudWatch should cover bare-metal instance status checks, network throughput, and any NAT or endpoint changes, even though NAT is gone. Add synthetic checks for Prism Central login, a test VM, and the internal artifact server. Before the upgrade, record baseline NCC results and cluster performance. After each node, verify NCC, CVM up status, storage rebuild completion, and VM availability before proceeding. After the full upgrade, run a test VM migration, a snapshot restore, and a Prism Central backup. Keep the artifact server and LCM configuration documented so the next CVE does not become another emergency.