Synchronous replication stall takes the primary down without a failure

FIRST 2 (pg-b, pg-c) asks for acknowledgements from two standbys, and synchronous_commit = on means the acknowledgement is a WAL flush. The set of synchronous standbys is the listed standbys that are currently connected, but the count requirement is hard: with two names listed and a required count of two, the primary will wait until both have flushed. There is no timeout on that wait - wal_sender_timeout governs the liveness of a replication connection, not the commit wait.

The chain that removed pg-b was: the 40-hour BI query plus max_standby_streaming_delay = -1 allowed replay to be delayed indefinitely; replay lag prevented the standby from recycling WAL segments; the volume filled; pg-b could no longer write received WAL, so it stopped sending flush acknowledgements. Once that happened, the required count of two could not be met, so every commit on pg-a blocked in the sync rep wait - visible as sessions waiting on SyncRep in pg_stat_activity and as a pg_stat_replication row for pg-b with a fixed flush_lsn.

This is the classic availability trap of a count-based sync configuration: durability was never at risk, availability was lost because a single, non-critical standby became mandatory for every write. hot_standby_feedback then makes the same query a bloat source on the primary, which is a secondary problem to solve later, not now.

The primary is alive, consistent and simply blocked, so the goal is to break the stall, not to fail over.

1. Freeze the blast radius first: on the primary, ALTER SYSTEM SET synchronous_standby_names = 'ANY 1 (pg-b, pg-c)'; SELECT pg_reload_conf(); takes effect for new commits without a restart, and writes immediately resume against pg-c alone. If you are not comfortable editing sync rep mid-incident, cancelling the BI query is the single action that unblocks everything, but it leaves the config trap in place.
2. On pg-b, SELECT pg_cancel_backend(pid) - or pg_terminate_backend if it will not cancel - for the 40-hour query, then confirm pg_last_wal_replay_lsn() advances and segment recycling resumes.
3. Recover disk deliberately: extend the volume or remove genuinely unnecessary files. Never delete files in pg_wal by hand on an active standby.
4. Confirm pg_stat_replication shows pg-b back in sync or potential state and that the sync set is what you intended.
5. Post-incident, take a fresh base backup of pg-b if you had to interfere with its data directory.

What EFM must not do: promote pg-c. The primary is up and serving reads; an EFM auto-failover triggered by write latency or by a primary-check timeout would create a split brain with the still-running pg-a and move the VIP out from under live sessions. Verify auto.failover behaviour and EFM's primary-liveness check before the next incident, because a well-meaning agent can turn this into data loss.

Three independent changes, each addressing a different link in the chain.

1. Sync rep semantics: synchronous_standby_names = 'ANY 1 (pg-b, pg-c)' with synchronous_commit = on (or remote_write if commit latency matters more than standby durability). ANY 1 keeps a real durability guarantee on at least one standby but degrades gracefully - losing either standby does not stop writes. If a single-standby outage must never block commits, that is the price: you are accepting that at the moment of a primary crash you may have one acknowledged copy, not two.
2. Standby replay policy: set max_standby_streaming_delay = '30s' and stop using -1. Give the BI workload its own route - a delayed replica that nobody depends on, or better, run reports off a snapshot instance - instead of holding replay hostage on a node that participates in durability. If hot_standby_feedback must stay on, keep the delay bounded so the feedback window is bounded too; otherwise the primary accumulates dead tuples that no vacuum can remove.
3. Headroom and alerting: alert on standby disk above 80% and on replication lag, and set max_slot_wal_keep_size if any slots exist, so a stalled consumer cannot silently fill the volume. Track pg_stat_replication.flush_lsn and the replay delay on each standby separately, because this failure was one standby's problem that became the primary's problem.

Rehearse it: a game-day test that fills a standby's volume on purpose tells you whether writes really continue, and whether EFM stays out of the way.