Backup window overrun, an 82 percent repository, and an RTO nobody has tested

Do the arithmetic first. 1.1 TB at 36 MB/s is 1.1e6 MB divided by 36, about 30,500 seconds, or roughly 8.5 hours. That matches the observed 06:35 to 06:48 finishes. A 1 GbE link delivers about 110 to 118 MB/s at best, so 36 MB/s is roughly a third of the link - something is throttling: a single NFS stream, small rsize/wsize, sync exports, or the dedupe appliance's ingest rate. Even at full line rate, 1.1 TB takes 2.6 hours, which would fit.

Rank by what it buys.

1. Stop sending a full logical database dump every night. Enable archive_mode and WAL archiving, take a weekly pg_basebackup, and archive WAL continuously. Nightly volume for the database drops to the WAL volume, and RPO improves from 24 hours to minutes. This is the largest structural win and it costs no hardware.
2. Convert the file share to incremental-forever with periodic synthetic fulls. A 2 percent daily change rate on 6 TB is about 120 GB per night. Nightly volume becomes roughly 200 GB total, or 1.5 to 2 hours at current throughput.
3. Fix throughput. Parallel streams, larger NFS read/write sizes, and checking the appliance's ingest ceiling. This turns 8.5 hours into 3 at the same data volume and is complementary to 1 and 2, not a substitute.
4. Capacity. An 82 percent full repository with 12 monthly fulls and 1.2 TB per month growth has roughly six months of runway; incremental-forever plus synthetic fulls also removes most of the duplicated full copies, so changes 1 and 2 solve the capacity problem that a second appliance would have solved with money.

Do not simply widen the window: 06:40 already collides with the 09:00 batch, and moving the boundary just relocates the collision.

Measure recovery, not backup. The RTO clock starts at 'we need the data' and stops when the service is serving. So the test is: declare a scenario, start a stopwatch, restore into an isolated VLAN or namespace, apply WAL to a chosen point in time, run consistency checks, then bring the application up against the restored database.

Steps:

1. Pick a target recovery point, for example 14:32 last Wednesday, so that WAL replay is genuinely tested rather than restoring the latest base backup.
2. Restore the base backup plus WAL to that timestamp, then run pg_amcheck and a row-count comparison against a known reference.
3. Start the application read-only against the restored copy and confirm it responds.
4. For the file share, restore a random 100 GB sample and compare checksums against the source manifest, then restore one deliberately deleted directory end to end.
5. Record actual elapsed time at each stage, the realised RPO (time of last recoverable transaction versus the target), and every manual step a responder had to improvise.

What it will probably reveal. A 7.2 TB full restore at 36 MB/s is about 55 hours. Even just the 1.2 TB database is about 9 hours at that rate, which already breaks a 4-hour RTO - and restores from a dedupe appliance are frequently slower than backups. The honest conclusion is that the RTO is currently unachievable for a full restore, so either the database needs a streaming replica or warm standby to meet 4 hours, or the tiering must be renegotiated, or restore throughput must be raised substantially, which is the case for the 10 GbE budget. Document this as a finding with numbers rather than reporting a green backup dashboard.

Also verify RPO mechanically: check pg_stat_archiver for failed or missing WAL segments and confirm there are no gaps in the archive over the last 30 days.

Job exit code is the weakest possible signal, and here it was green while the window overran and the repository filled.

Add these alerts.

• Job duration above 5 hours, and duration trending upward week over week, so the overrun is caught at 4h30m rather than at 06:48.
• Bytes transferred deviating more than 30 percent from the trailing seven-day average in either direction - a sudden drop usually means partial data, not efficiency.
• Snapshot age over 26 hours for every protected object, tracked per client, so one silently failing host is visible.
• Repository free space below 20 percent, and a projected-full date derived from the monthly growth rate.
• WAL archiving failures: archive_command non-zero, last_failed_wal non-empty, and any gap in the archived sequence.
• Appliance health: ingest errors, failed dedupe jobs, and filesystem errors on the repository.

Add verification that exercises restore. A nightly canary that restores a fixed 5 GB dataset from last night's snapshot into a scratch namespace and validates checksums proves the chain end to end without a full drill. Run it from the same infrastructure a real restore would use, so credential and network problems surface nightly rather than during an incident. Weekly, restore one randomly chosen database table and compare a checksum against production.

Run the full documented fire drill quarterly with a stopwatch and a named responder, and keep the last three reports alongside the RTO target. If the drill shows the target is not met, the finding belongs in the risk register, not in a footnote.