Backup window overrun versus the dispatch window
Meridian Freight runs a nightly backup from 01:00 to 05:00 across 340 servers. For six weeks the job has finished between 06:40 and 07:20, overlapping the 06:00 dispatch window when the order-routing platform needs full disk I/O. Dispatch staff report lookup timeouts. The dedupe appliance sits at 92% write-cache utilisation for the whole run; network links average 40% utilisation. A vendor quotes £180k and 14 weeks for a second appliance. The DBA offers to move the two largest databases to weekly full plus daily incremental, cutting backup volume by roughly 35% but stretching their restore time from 2 hours to 9. The change board meets in 9 days.
1. What do you put to the change board in 9 days, and what evidence has to be in front of the board for the decision to be defensible?
Read the worked answer
Frame the constraint first
The board should not be asked to choose between £180k and a 9-hour restore until it is clear which resource is actually saturated. Two measurements point in different directions: the appliance cache is at 92% (saturated) while the network averages 40% (not saturated). That strongly suggests the job is bounded by cache ingest, not bandwidth - but averages hide peaks, so confirm with per-stream throughput and cache utilisation sampled at, say, 5-minute intervals across the whole run.
Evidence to table
- p95 job duration over six weeks, not the mean, broken down by phase (which job phase overlaps 06:00).
- Dispatch latency (p95 lookup time) correlated against those same phases - this is what converts an IT metric into a business cost.
- The service owner's signed restore-time requirement for the order-routing databases specifically, as distinct from the other 338 servers.
- Cost per night of degraded dispatch, so the options can be compared on one axis.
Recommendation to bring
A low-cost, reversible change now (retime or rate-limit the streams so the appliance cache stays below saturation, and move the order-routing volume to a separate snapshot taken outside the dispatch window), with the appliance purchase held pending the evidence. Do not let the DBA's 35% reduction become the plan until the restore-time question is answered - it trades a nightly problem for a rarer but much larger one.
2. The service owner replies that a 9-hour restore is unacceptable and therefore the DBA's proposal is off the table. How do you respond?
Read the worked answer
Agree with the conclusion only if the premise is right. A restore objective is a business property, not a technical one, so the useful question is: 9 hours to what?
Separate two things that are usually conflated:
- Routine recovery (a dropped table, a bad deployment) - this can be served by a snapshot on primary storage or a short incremental chain, measured in minutes.
- Disaster recovery of those two databases - this is where the 9 hours lands.
If the service owner genuinely needs 2 hours for disaster recovery as well, then the 35% volume reduction is not available and the appliance purchase becomes the route - but now it is justified against a stated objective rather than against a vendor quote. Ask for the requirement to be written down and owned; that single artefact decides the question and prevents the argument recurring.
If only the routine case matters, accept the DBA's proposal and keep the appliance decision open.
3. Suppose only one action can be funded this quarter. How do you choose?
Read the worked answer
Test which resource is the binding constraint, because only the binding one buys you time.
- If job duration scales with bytes written and cache utilisation tracks it, more cache capacity shortens the run - the appliance is the funded action.
- If duration scales with file count or metadata operations while cache is high but not the limit, a second appliance buys nothing and the answer is restructuring what the job backs up (fewer, larger streams; exclusions; changed schedules).
- If the network had been the saturated resource, neither answer would have worked.
The trap is treating 92% cache as proof by itself. It is a strong signal, not a diagnosis. Fund whichever action the bottleneck evidence supports, and spend the near-zero-cost scheduling and rate-limit changes this week regardless - they are reversible and buy the time needed to gather the evidence.