I85PREP
⌘ K
Citrix CCE-N study materials
Citrix CCE-N study resources

Nightly HA Failover and an Upgrade With Asymmetric Crypto

Citrix CCE-N · Put your knowledge to work in context.

All case studies

Nightly HA Failover and an Upgrade With Asymmetric Crypto

A pair of MPX 8910s runs active/standby high availability in an insurance data centre. Heartbeats use interface 1/1, a trunk that carries client VLAN 100 to the same access switches as production traffic. Failover on NIC failure is enabled on every interface, and preemption is enabled because a previous engineer wanted the primary to always be primary. Every night around 02:10 a storage backup saturates the 10G link, the network team observes LACP on 1/1 reset, the pair fails over, the backup job dies, and at about 02:25 the pair fails back. Users see 20-30 second outages and ICA sessions drop. The pair is also being prepared for an upgrade from 13.1 build 42.47 to a newer build, and the secondary node has one SSL chip while the primary has two. Citrix ADM onboards the pair by the primary NSIP and shows a gap in HDX Insight data every night.

1. Walk through the diagnosis. Which commands and log evidence would confirm the root cause, and how would you exclude the alternatives?

Read the worked answer

Start by proving the failover reason rather than the failover itself.

Commands: show ha node and show node for peer state, sync state, and the recorded failover reason; stat ha for failover counters; show interface summary and stat interface for drops, errors, and flap counts on 1/1; show channel or the LACP statistics for the trunk; show route monitor for any monitor-driven failover; and ns.log filtered around 02:10 for HA, NIC, and failover messages.

Confirming the hypothesis: the ns.log timestamps for the interface-down event should precede the failover entry, and the failover reason should reference a NIC or heartbeat failure rather than a monitor. Correlate with the LACP reset on the access switch and with the backup job's start time.

Alternatives to exclude:

  • Heartbeat loss without link loss would show as an HA communication failure, not a NIC failure; check whether heartbeat packets are being dropped by congestion or QoS on the trunk.
  • Route monitor failure would show the upstream monitor going DOWN in the same window.
  • Resource exhaustion on the primary (memory, SSL card, connection table) would appear in stat output before the event.

The 02:25 failback is explained by preemption: as soon as the failed node's interface recovers, the configured primary takes over again, doubling the outage.

2. Design the fix so this cannot recur, and explain which changes carry risk.

Read the worked answer

The core problem is that production traffic, LACP, and HA heartbeats share one failure domain, and that every interface is allowed to trigger a failover.

Changes, in order of value:

  1. Separate the heartbeat. Move HA communication to a dedicated interface, ideally direct-attached between the two appliances or on a dedicated non-blocking VLAN with no LACP, no QoS, and no backup traffic. This removes the shared failure domain entirely.
  2. Reduce failover triggers. Use set interface <id> -haMonitor OFF on interfaces whose failure does not mean the node cannot serve traffic, and keep HA monitoring only on interfaces that genuinely indicate loss of service. Disabling the monitor makes the interface invisible to HA, so document each one.
  3. Replace NIC monitoring with path monitoring where appropriate. A route monitor that tests the upstream next hop reflects the user experience better than a link state, and it does not react to a brief renegotiation.
  4. Disable preemption, which is off by default, and fail back deliberately during a window.
  5. Stabilise the trunk. Set longer LACP timers, and move or rate-limit the backup traffic so it is not competing on the same links. If the backup must traverse the ADC, shape it rather than letting it saturate 10G.

Risk: removing HA monitoring from an interface means a genuine failure there will not trigger failover; compensate with route monitors and monitoring alerts. Changing heartbeat topology requires a maintenance window, since HA must be re-established and re-synced.

3. Sequence the upgrade, address the asymmetric SSL chip, and explain the nightly HDX Insight gap in ADM.

Read the worked answer

SSL chip asymmetry first. Two crypto chips versus one means the standby cannot carry full load after a failover, and differing crypto capability can also cause TLS 1.3 ECDSA behaviour to diverge between nodes. Either fit the second card or formally accept reduced post-failover capacity and document it in the runbook; do not upgrade on top of an unquantified capacity gap.

Upgrade sequence: take a backup and save the configuration; confirm show ha node reports sync enabled; put the site out of GSLB or schedule the window; upgrade the node that is currently secondary (it is not serving traffic), reboot, and verify it rejoins as secondary and synchronises cleanly; force a failover during the window, accepting that in-flight connections on the old primary are dropped; upgrade the former primary; verify sync, then fail back if required. Check both nodes report the same build before declaring done.

The HDX Insight gap: AppFlow must be configured on both nodes, not just the primary, and ADM should be onboarded once, using the pair's NSIP, so ADM discovers the peer and treats the pair as a single instance. Onboarding both NSIPs separately produces duplicate instances and double counting. A short gap at failover is expected because AppFlow records buffered on the failing node are lost, but a 15-minute gap points to something else: AppFlow not running on the new primary, the collector or analytics profile not bound there, or clock skew. Verify NTP on both nodes and ADM, check stat appflow on the surviving node, and confirm flow records resume before closing the change.