Minor Version Patch Halt Across a PostgreSQL Fleet
A DBA team manages 30 PostgreSQL 13.7 databases with NDB. They create a new software profile for PostgreSQL 13.10 and start a patch job against all 30 VMs in one batch during a six-hour maintenance window. Four databases are streaming replicas; one primary is 2 TB and has a 45-minute backup window. Several databases use PostGIS and pg_stat_statements. The job patches 10 databases, then fails on the eleventh because /var/tmp has only 900 MB free and the PostGIS extension version is not compatible with the target binary. NDB marks the batch as failed and does not continue to the remaining 19 databases. The eleventh database is still running on 13.7, but its NDB agent is reachable. The team wants to know whether to roll back, resume, or restart the batch, and how to avoid the same failure in the next window.
1. Was running all 30 databases in one patch batch appropriate? Explain the risks and the correct batching model.
Read the worked answer
No. A single batch across 30 databases maximizes blast radius and removes the ability to learn from an early failure. It also ignores differences in database size, criticality, replication role, extension usage, and backup windows. The 2 TB primary and the four streaming replicas should not be treated like small standalone databases.
A safer model is a canary wave followed by controlled waves. Start with one or two low-criticality, non-replicated databases that use the same extensions. Validate the NDB agent, database startup, extension behavior, application smoke tests, and backup/log continuity. Then patch in groups by application tier or criticality. For replicated databases, decide explicitly whether to patch replicas first or to use a controlled failover; do not patch a primary without a tested rollback or failover path. Keep each wave inside the maintenance window and leave buffer time.
NDB patch jobs are not a substitute for a change plan. Scope the job to the wave, verify prechecks for every database, and stop the rollout when a wave fails. The correct response to the eleventh-database failure is not to restart all 30 blindly; it is to isolate the failing database, remediate it, and resume the remaining databases in smaller groups.
2. Diagnose the failure and prescribe the remediation path for the eleventh database and the remaining 19. Include validation and rollback criteria.
Read the worked answer
The failure has two independent causes. The 900 MB /var/tmp is too small for the patch staging and installation work, and the installed PostGIS version is not compatible with the PostgreSQL 13.10 binaries. The agent being reachable only proves the control path is alive; it does not prove the database is ready to patch.
Remediate the eleventh database by freeing or extending temporary space, confirming the patch tool's staging requirements, and checking the PostGIS compatibility matrix for the target PostgreSQL minor version. Upgrade or adjust the extension only with an approved procedure, or exclude this database from the wave. Then patch it individually and validate SELECT version(), extension versions, application queries, replication state if applicable, NDB agent heartbeat, and backup/log continuity.
For the remaining 19, do not resume the failed batch as-is. Re-run prechecks across the group, remove any other database with the same extension or space profile, and schedule smaller waves. Rollback should be reserved for a database that fails after patching and cannot be stabilized. Because a minor-version rollback through restore can lose post-patch changes, prefer fixing forward when the database is healthy; if rollback is required, restore from the pre-patch snapshot or time machine to a new database, validate, and switch connections deliberately.