Control-Plane Maintenance on a Shared NDB Server
An enterprise runs NDB 2.x on a six-node AHV cluster. The NDB server VM is hosted on node C and manages 120 database VMs, mostly PostgreSQL and MySQL. Each database has a time machine with a 4-hour snapshot SLA and 14-day retention. The NDB metadata database resides on the NDB server VM, and its volume is 68% full. During a planned AHV upgrade, node C must be evacuated and rebooted. While node C is down, database clients remain connected, but the NDB UI becomes unavailable, provisioning and snapshot jobs fail, and several database agents report stale heartbeats. After node C returns, most jobs resume, but two time machines miss their SLA window, and one clone refresh remains in a failed state. The storage team also reports that the metadata volume grew 12% in the last month. The platform owner asks whether NDB should be treated like a stateless management tool or a protected workload.
1. Explain which failures in the scenario are expected control-plane failures and which indicate a design or capacity problem. What dependencies should the team document?
Read the worked answer
Expected control-plane failures include loss of the NDB UI and API, failed provisioning, failed snapshot orchestration, failed clone refresh, and stale agent heartbeats. The database data plane can continue because existing database VMs and their storage do not require the NDB server for an open client connection. That does not mean the outage is harmless: snapshot SLAs, log backups, patching, and clone operations are all control-plane actions.
The design problems are different. A single NDB server on a node that must be evacuated, with no HA or management-cluster placement, turns routine maintenance into a control-plane outage. The metadata volume at 68% with 12% monthly growth is a capacity risk; at that rate it approaches 90% in roughly two months. The failed clone refresh and missed SLA windows are also not self-healing; they need an operator to reconcile state.
The team should document these dependencies: NDB server to metadata database, NDB server to agents over the management network, DNS and NTP for all components, Prism Element/Prism Central access, AHV storage containers, and the backup/log path. It should also record which operations are data-plane versus control-plane, and define RPO/RTO for the NDB server itself.
2. Recommend a target architecture and a pre-maintenance runbook for this environment. Include how to handle metadata growth and missed snapshots.
Read the worked answer
Treat NDB as a protected management workload. If the deployed release supports NDB server HA, use it; otherwise place the NDB server on a dedicated management cluster, apply anti-affinity so it does not share a node with critical databases, and protect the NDB server VM with a VM-level backup or protection policy. Size and monitor the metadata volume separately from database storage.
Before maintenance: confirm no active provisioning, patching, snapshot, or clone-refresh jobs; verify all agents are reachable; take an on-demand snapshot of databases that cannot miss their SLA; export or record NDB configuration; and validate that the NDB server backup or metadata backup is current. During the outage, do not expect queued snapshots to be replayed automatically.
After maintenance: check agent heartbeats, rerun failed jobs deliberately, take manual snapshots for the two time machines that missed their SLA, and confirm that log backup chains are continuous. For metadata growth, expand the volume before the next window, purge stale time machines and expired clones according to retention, and set alerts at 70%, 80%, and 90%. Forecast growth using the observed 12% monthly rate rather than waiting for the volume to fill.