EDB PostgreSQL Associate study materials
EDB PostgreSQL Associate study resources

Synchronous replication that takes the site down

EDB PostgreSQL Associate · Put your knowledge to work in context.

All case studies

Synchronous replication that takes the site down

A primary runs with synchronous_standby_names = 'FIRST 2 (sb1, sb2)' and synchronous_commit = on. sb1 is in the same rack; sb2 is in a second datacenter with 30 ms round-trip latency. During a switch maintenance window sb2's link flaps for 90 seconds. Every INSERT and UPDATE in the application hangs for the duration and then fails with statement timeouts; pg_stat_activity shows sessions waiting on SyncRep. As soon as sb2 returns, everything resumes normally. The CTO wants both zero data loss on failover and no write outage when any single standby is unavailable, and asks whether setting synchronous_commit = off during maintenance windows is a sensible mitigation.

1. Explain the outage and assess whether the CTO's two requirements can both be met with two standbys.

Read the worked answer

FIRST 2 (sb1, sb2) means the primary waits for acknowledgement from two synchronous standbys on every commit. With sb2 unreachable the quorum can never be satisfied, so each commit waits indefinitely - there is no commit timeout; wal_sender_timeout governs silent replication connections, not the client's wait for SyncRep. Statement timeouts eventually kill the client sessions, not the replication wait.

The two requirements conflict at this topology size:

  • Zero loss on failover requires at least one standby to have flushed the commit before the client is told 'committed'. That is a quorum of size ≥ 1.
  • No outage when one standby is down requires the quorum to be satisfiable by the surviving nodes.

With two standbys you can have availability (ANY 1) or a two-copy guarantee (FIRST 2), not both. Concretely:

Setting Survives one standby loss? Data loss risk on failover
FIRST 2 (sb1, sb2) No - writes stall None (two copies)
ANY 1 (sb1, sb2) Yes Small: if sb1 acked and the primary dies before sb2 catches up, promoting sb2 loses recent commits
FIRST 1 (sb1) No - rack loss stalls writes None, if sb1 survives
Third standby in a third zone, ANY 2 (sb1, sb2, sb3) Yes None

Recommendation: ANY 1 (sb1, sb2) now, plus a third standby in a third failure domain with ANY 2 once it exists. With ANY 1, the standby that acknowledged is by definition the one with the data - so failover must target the acker, not simply the nearest node.

2. Is `synchronous_commit = off` a safe mitigation for maintenance windows? What would you do instead?

Read the worked answer

No. synchronous_commit = off drops the durability guarantee entirely rather than relaxing quorum:

  • The client is told 'committed' before the local WAL is flushed. A primary crash can lose up to roughly three times wal_writer_delay of committed transactions (a few hundred milliseconds by default).
  • It also stops waiting for standbys, so you have reduced the loss window on paper while increasing the risk of losing transactions that the application believes are durable.
  • Worse, it is a session or cluster-level setting that is easy to leave switched on after the window closes.

Better options, in order:

  1. Change the quorum, not the durability: synchronous_standby_names = 'ANY 1 (sb1, sb2)' is reload-only (SIGHUP), takes effect immediately, and keeps a real acknowledgement guarantee. This is the correct maintenance-window switch.
  2. Set per-session: if a batch job must not block, SET synchronous_commit = local in that session only, so its commits are locally flushed but not replicated - loss is bounded and scoped, and normal OLTP keeps its guarantee.
  3. Set wal_sender_timeout sensibly so a dead walsender connection is detected and removed from the sync set quickly, rather than the primary holding a stale slot.
  4. Never disable the guarantee cluster-wide to work around a maintenance event; fix the quorum instead.

3. The primary dies while sb2 is still catching up. How do you decide which node to promote, and what do you do first?

Read the worked answer

Before promoting, fence. Revoke the primary's VIP, STONITH it, or otherwise guarantee it cannot come back and accept writes. Split-brain costs far more than a few lost transactions and is not recoverable by re-running a command.

Then establish which node has the most committed data:

  • On the survivors: SELECT pg_last_wal_replay_lsn(), pg_last_xact_replay_timestamp(); on each. The higher replay LSN wins; the timestamp confirms how far behind the other is.
  • If the old primary is merely unreachable and you can read it, pg_stat_replication shows flush_lsn and replay_lsn per standby - the node whose flush_lsn matched the primary's last commit is the safe one.
  • With ANY 1 in force, the node that was acknowledging is the one guaranteed to hold every commit the clients saw. That is the candidate, even if it is the higher-latency datacenter node. Choosing the closer node because it is more convenient is how you lose data.

Then: pg_promote() (or pg_ctl promote), repoint the application, and verify pg_is_in_recovery() is false and the new primary accepts writes. Rebuild the old primary with pg_rewind or a fresh base backup rather than trying to reattach it as-is - an un-fenced old primary rejoining as a standby with a diverged timeline must never be trusted blindly.

Finally, record the actual RPO achieved by comparing the last transaction the application logged against what the new primary contains, and feed that measurement back into the quorum decision.