HA Failover During a Batch Window: Choosing the Safer Path
A CX 10000 HA pair protects stateful east-west traffic. During a batch window, the active node reports a degraded DPU and the pair is considering a failover to the standby. The batch job is mid-run with hundreds of thousands of active stateful sessions. The team notes that a failover will reset stateful sessions unless state sync is healthy, and the last health check of state sync was inconclusive. The change window closes in 20 minutes.
1. Should the team fail over now? Walk through the decision.
Read the worked answer
Do not fail over blindly mid-batch when state-sync health is unknown.
Decision factors:
- Is the degradation service-affecting? If traffic is still flowing and the degraded DPU is not dropping sessions, a forced failover may cause more harm (mass session resets) than waiting.
- State-sync status: if state sync is unhealthy, failover = cold standby = all in-flight sessions reset. Confirm sync health first; if it cannot be confirmed in the window, assume resets.
- Blast radius: hundreds of thousands of active sessions resetting mid-batch could corrupt or fail the job.
Safer path: stabilize the active node, let the batch complete, then perform a controlled failover in a planned window with state sync verified and the job paused. Failover is a tool, not a reflex; the goal is least disruption, not fastest action.
2. What would you verify before any controlled failover, and how do you confirm success afterward?
Read the worked answer
Pre-failover checks:
- State-sync health: confirm the standby is receiving session state and its table is populated; a cold standby will reset sessions.
- Standby readiness: DPU health, policy version match, and that the standby can take the full session load.
- Traffic quiesce: pause or checkpoint the batch job so a reset is recoverable.
- Rollback plan: ability to fail back if the standby underperforms.
Post-failover confirmation:
- Session continuity: sample active flows to confirm they survived (or quantify resets).
- Counters: new active node shows expected session counts and no flow-table pressure.
- Application: batch job resumes without errors; retransmit/latency return to baseline.
- Health: both nodes report healthy; state sync re-established to the new standby.
Document the reset count so the impact is known, not assumed.