Three-Node Cluster Upgrade with a Broken NTP Path and an Expiring Certificate
A three-node Secure Workload cluster (one master, two worker nodes) in a primary data center reports 1,240 agents, 38 percent disk utilization and healthy replication. You must upgrade from the 3.7 release train to 3.9 inside a six-hour window. Known facts: worker-2 cannot reach the external NTP servers because the management firewall permits only DNS and the UI ports outbound; DNS resolves correctly from all three nodes; SMTP alerts are configured to an internal relay; two AWS VPCs are onboarded through a Secure Connector; and the cluster's web certificate expires in 45 days. Compliance requires that no workload loses telemetry for more than 30 minutes, and the security operations team relies on the flow history for investigations, so losing collection is not acceptable. The previous upgrade, done by another team, left one node reporting a time skew that took two days to notice, and agents on 60 hosts stopped reporting until they were manually restarted.
1. What is your pre-upgrade check sequence, and in what order do you expect the nodes to be upgraded? Justify the order against the 30-minute telemetry requirement.
Read the worked answer
Check in this order, because each item can block the upgrade or silently degrade it after.
- Cluster health and replication: if a node is already degraded, the upgrade converts a soft failure into an outage. Confirm all nodes are healthy, in sync, and that no re-sync is in progress.
- Time: compare time on all three nodes and verify NTP synchronization. Do this before anything else because a skewed node can fail certificate validation and produce misleading errors during the upgrade.
- Disk: 38 percent is comfortable, but confirm the post-upgrade retention needs; an upgrade that leaves the cluster near-full makes the next month's flow retention unpredictable.
- External services: DNS, NTP, SMTP, and syslog reachability from each node individually, not just from the master. Worker-2's blocked NTP is a real finding.
- Certificate: a certificate expiring in 45 days is a pre-upgrade task, not a post-upgrade one. Replace it while you have a change window, and confirm the agents and connector trust the new chain - otherwise you upgrade into a trust failure.
- Configuration backup and documented rollback point.
On order: expect the non-master nodes to be upgraded before the master, with the cluster orchestrating the sequence so that the master remains available to serve configuration and receive agent data. That is what protects the 30-minute requirement - agents buffer locally and reconnect to whichever node is available, so you never take the whole control plane down at once. Do not upgrade nodes in parallel or offline the master first; that maximizes the collection gap and can force agent re-registration.
2. Worker-2 has been running with an unreachable NTP source. Explain why this matters for Secure Workload specifically, how you would fix it inside the window, and what verification proves it is fixed.
Read the worked answer
Secure Workload is a time-series system: flows, ADM conversations, policy compliance evaluation and forensic queries are all bucketed by time. If worker-2's clock drifts, three things go wrong. First, flow records ingested through that node land in the wrong time buckets, so a forensic query 'what did host X talk to between 02:00 and 03:00' silently returns incomplete results - the worst failure mode, because the answer still looks plausible. Second, ADM windows and policy simulation compare against timestamps, so skewed data distorts which flows are considered in-window. Third, TLS and internal service authentication can fail when clock skew exceeds certificate tolerance, which is consistent with the earlier incident where agents stopped reporting.
Fix it inside the window by pointing all three nodes at internal NTP sources that the management firewall already permits, rather than opening UDP 123 to the internet. Open the rule for the internal sources only, apply the configuration, and restart or re-sync the time service on worker-2.
Verification is not 'the command returned no error'. Confirm: all three nodes report the same stratum and a small offset; the cluster health page shows nodes synchronized; a test flow generated from a known host appears in flow search with a timestamp matching the host's own clock to within a second; and ADM shows recent conversations rather than a gap. Recheck 24 hours later, because drift is a slow failure.
3. After the upgrade, what do you validate before declaring the window closed, and what would make you roll back or escalate?
Read the worked answer
Validate in the order the business cares about: telemetry continuity, then policy correctness, then integrations.
Telemetry: confirm the agent count matches the pre-upgrade baseline (1,240), not just 'agents are connected'. Compare per-scope counts; agents can reconnect while a subset stays stale. Check the largest gap in flow ingestion per node and confirm it is under 30 minutes. Check whether agents were auto-updated to the matching agent version and whether any are held back by an incompatible OS - the earlier incident suggests 60 hosts that silently stop reporting, so look for hosts with no flows in the last hour rather than hosts marked disconnected.
Policy: confirm the enforced scopes show the same enforcement mode and rule counts as before the upgrade, and that policy compliance is still being evaluated. An upgrade that resets a scope to monitor mode is a silent security regression.
Integrations: verify the Secure Connector to both AWS VPCs is up and the connector's inventory matches, that SMTP alerts actually deliver a test message, and that the renewed certificate is the one being served.
Escalate or roll back if: the master fails to come up healthy, replication does not converge, a sustained telemetry gap exceeds the 30-minute limit, or agents fail to re-establish after a reasonable buffer period. Rolling back a partially upgraded three-node cluster is not trivially safe, so the decision point should be defined before the window starts, and the trigger should be based on observed data rather than elapsed time.