Cluster Replication Divergence Before a Partner Cutoff
Your PingFederate cluster has four nodes. Nodes 1 and 2 run 11.3.0; nodes 3 and 4 run 11.3.1. The load balancer sends runtime SSO traffic to all four nodes, but administrators connect to the admin console only on node 1. Last week an administrator created a new SAML SP connection on node 2. The connection works on node 2, but it is missing from nodes 1, 3, and 4. Node 4 also still presents an old metadata file for an existing partner. There is no planned maintenance window, and the partner cutoff is in 72 hours. Cluster status shows all nodes as running, but the replication log has warnings about clock skew and version mismatch. You need to get configuration consistent before changing certificates.
1. Identify the likely causes of the divergence and give a safe remediation order. Explain why you would not simply recreate the connection on node 1.
Read the worked answer
The version mismatch and clock skew are the red flags. PingFederate cluster replication expects compatible versions and synchronized clocks; mixed 11.3.0 and 11.3.1 nodes can produce partial or failed replication, and the warnings must be treated as a cluster health problem, not a single-connection problem. Recreating the connection on node 1 while replication is broken would create two divergent configurations and increase the risk of losing one of them.
Safe order:
- Stop configuration changes.
- Back up the configuration and export the working connection details from node 2.
- Correct clock synchronization on all nodes and verify skew is within tolerance.
- Bring all nodes to the same supported patch level.
- Use the supported cluster resync or replicate-configuration operation from an authoritative node, or remove and rejoin node 4 if it remains stale.
- Verify the connection and metadata on every node, then proceed with certificate work.
Do not copy configuration files between nodes.
2. Would using the PingFederate admin API to push the missing connection solve the problem? What validation would you perform before the 72-hour cutoff?
Read the worked answer
The admin API is not a bypass for a broken cluster. Writes through the API replicate like console writes, so pushing the connection while replication is unhealthy can create more divergence. Use the API for read-only inventory or for an idempotent change after cluster health is restored. If the cutoff cannot move, fix replication or isolate one node with a documented exception; do not run two unsynchronized admin authorities.
Validation before cutoff:
- Confirm all four nodes show the same connection, metadata, certificate, and adapter configuration.
- Run test SSO through each node's runtime address, not only through the load balancer.
- Check cluster status and replication logs for errors and lag.
- Inspect audit and server logs on each node for signature or metadata errors.
- Confirm the partner has the correct metadata and certificate before changing certificates.
If any node disagrees, stop and resolve cluster health before the partner change.