Encryption Enabled, Single KMS, and a DR Site That Never Got the Keys
A financial services firm runs a 12-node AHV cluster on self-encrypting drives with data-at-rest encryption enabled at build time. Keys are held by a single external KMIP appliance in the primary data centre - one instance, no HA. A 6-node DR cluster has self-encrypting drives but encryption was never enabled; 60 protection domains replicate to it asynchronously. Compliance requires key escrow, annual key rotation, and survival of total primary-site loss. During a KMS maintenance window last month, a planned reboot of one node failed to unlock its drives and the node stayed out of service for 90 minutes. A test failover of an encrypted VM to the DR site also failed.
1. Explain why the node failed to rejoin and what you would say to a colleague who proposes 'just disabling encryption on that node' to get it back.
Read the worked answer
What happened. With self-encrypting drives, the drive stays locked until it receives its key. A running host caches what it needs in memory, which is why the VMs on other nodes kept running during the maintenance window - the failure only appears at the moment a node needs to re-fetch keys, i.e. on reboot. A single, non-HA KMS plus a maintenance window is a scheduled outage of the ability to boot nodes.
Why 'disable encryption on that node' is not an option: encryption is a cluster-level property of the drives and the key path, not a per-node toggle. There is no supported way to un-encrypt one node to bring it back; attempts to force it either fail or create a data-integrity and compliance problem. It also moves the conversation from 'availability design' to 'break the control', which the auditors will find.
What actually fixes it:
- KMS as a redundant pair or KMIP cluster, in a fault domain that is not the Nutanix cluster it protects, reachable over a path that does not depend on that cluster.
- Monitoring on KMS reachability and on the certificate used for the KMIP connection - an expired client certificate looks identical to a dead KMS.
- A documented rule in the change process: no node reboots during KMS maintenance, and KMS maintenance requires an approved window with the platform team present.
- A rehearsed recovery procedure for 'KMS is gone' that covers both the boot path and the key escrow restore.
2. The DR cluster has self-encrypting drives but encryption was never enabled, and the encrypted VM failover failed. What is your decision and what are the prerequisites before the next test?
Read the worked answer
Decision: enable encryption on the DR cluster before any further replication testing, and treat DR key availability as part of the DR design rather than an afterthought.
The failure is expected: the encrypted source data has no key path on the target, so the target cannot unlock and start it. Encrypting only the primary site means the DR site is simultaneously unprotected and unable to recover the very workloads it exists for.
Prerequisites, in order:
- Enable encryption on the DR cluster and confirm the drives are correctly recognised and the key path works - this is a change with capacity and performance implications, so measure it now, not during a failover.
- Decide the key topology: either the DR site has its own KMS that receives replicated key material, or the DR hosts can reach a surviving KMS if the primary site is lost. 'Accessible only from the primary data centre' fails the total-site-loss requirement.
- Escrow keys independently of the cluster - with the KMS vendor or an offline escrow with split custody - and verify a restore actually yields usable keys.
- Write the unlock sequence into the DR runbook and put it in the correct order: KMS available, then cluster, then nodes, then VMs. Encryption adds a hard ordering constraint that a non-encrypted DR plan does not have.
- Re-test with the failure mode that matters: fail over, then reboot a DR node, because reboot is where key retrieval is exercised.
3. Compliance requires annual key rotation with no data loss. Outline the runbook and the acceptance checks.
Read the worked answer
Runbook structure:
- Pre-checks: KMS redundancy healthy, escrow current and verified restorable, cluster healthy, no pending node reboots, confirm the rotation method against the supported matrix for this AOS and KMS version before scheduling.
- Change window with the platform, KMS and application owners present; snapshot/backup taken and a restore point identified.
- Rotate at the KMS/KMIP layer according to the vendor's procedure, keeping the previous key material available until the new path is confirmed.
- Validate: cluster and all nodes healthy, every node able to unlock after a controlled reboot of one non-critical node, VMs running, replication to DR still functioning.
- Retire old key material only after the acceptance window closes.
Acceptance checks (the evidence the auditor will ask for):
- Rotation performed within the policy interval, with date, operator and approver recorded.
- Escrow updated to match the new key material and a restore test logged.
- Post-rotation DR test result, including a reboot at the DR site.
- No unplanned outages; any deviation documented as a change record.
Judgement note: rotation is the highest-risk routine task in an encrypted estate because every failure mode is 'the cluster cannot unlock'. Never combine it with an AOS upgrade or a DR test in the same window.