Choosing a recovery strategy before the peak-season freeze
Ardent Home is an online retailer with £95m revenue, 60% of it in November and December. Orders run through a SaaS order-management platform; stock is held and picked by a single third-party logistics provider in one leased warehouse. Order processing has a recovery time objective of eight hours. The current recovery arrangement is a spare server room in head office with weekly tape backups. Two designs are on the table: a warm standby in a second region at £310,000 a year, or rebuilding on the vendor's multi-region cloud service at £470,000 a year with continuous replication. Finance has asked for a recommendation before the peak-season freeze starts in ten weeks. The 3PL contract contains no continuity, notification or recovery obligations.
1. Evaluate the two technical options against the stated requirements and recommend one.
Read the worked answer
Start with what the requirement actually demands. The RTO is eight hours for order processing. The current arrangement cannot meet it: a tape backup with weekly restore is an RPO of up to seven days, which means losing up to a week of orders and customer records. Whatever option is chosen, the present position is not a degraded version of a solution - it is a material, unquantified data-loss exposure that has never been formally accepted.
Warm standby, £310k. Meets an eight-hour RTO only if the failover is rehearsed and the data replication interval is set to meet the actual RPO. Cheaper, but it is a second environment to keep patched, licensed and tested, and the annual cost understates the operational burden. Typically fails in practice because the standby drifts out of configuration parity with production.
Multi-region cloud, £470k. Continuous replication gives an RPO measured in minutes and meets the RTO with failover automation. Costs £160k more per year.
Recommendation: the cloud option, conditional on the replication, failover automation and at least two successful tests being complete before 1 November. Reasoning: the £160k differential is roughly 0.17% of revenue; a single day of lost November trading, at ~£156m of peak revenue over the period, dwarfs it. The RPO alone justifies the choice - the warm standby with an unrehearsed replication interval is a false economy.
But be explicit about the limit: this option recovers technology. It does not recover the warehouse, the 3PL relationship or the customer-service capacity. Technology strategy is not continuity strategy, and presenting it as such would be the classic error here.
2. How do you address the single 3PL warehouse while the contract offers no leverage?
Read the worked answer
Name the exposure plainly. A single third-party warehouse holding all stock, with no continuity obligations in the contract, is a greater threat to peak-season revenue than any technology failure. Yet the technology project is the one with a business case - which is exactly the misallocation the BIA should prevent.
Levers, in escalating order:
- Contract renegotiation at renewal or next variation. Minimum requirements: a supplier BIA and plan on request, defined notification time for disruption, a stated recovery time for picking and dispatch, a right to audit or to receive assurance evidence, and a right to invoke an alternate site. Certainty is worth little without these.
- Due diligence now. Ask for their plan, their last exercise report and their insurance position. A supplier with no plan and no exercise history is a risk you can quantify before it happens.
- Physical mitigation. Identify a second fulfilment option - a second site, a regional 3PL, or an overflow arrangement with a carrier. Even a limited alternate for the top 20% of SKUs changes the consequence materially.
- Joint exercising. Run one exercise with the 3PL before the peak, using a scenario where their site is unavailable for 48 hours. What they cannot do will surface quickly and cheaply.
- Transfer what can be transferred. Business interruption insurance, and a contractual remedy if renegotiation succeeds.
If none of that lands before peak, document the residual risk, quantify it in revenue terms, and escalate to the executive for acceptance with a date. That is a decision above your pay grade and should be recorded as such.
3. Implementation: what must be true before the organisation can credibly claim an eight-hour recovery capability?
Read the worked answer
A capability exists when it has been demonstrated, not when it has been bought. The gap between signing the cloud contract and holding a defensible eight-hour RTO is where most implementations fail.
Conditions to be met, in order:
- Documented procedures written for the people who will use them. Who declares an incident, who authorises failover, who tells customers, who reconciles orders lost in the gap. Technical failover with no decision rights attached produces a recovered system nobody is running.
- Roles and competence. Named alternates for the failover decision, out of hours, including during peak when the team is already stretched. Train them, then sample them.
- Replication verified against the RPO. Measure the actual lag under peak load, not under test conditions. Peak is the only load that matters here.
- At least one full failover test before 1 November, followed by a failback test. Failback is the step that is always skipped and always causes the second outage.
- Dependency coverage accepted or documented. The eight-hour claim covers order processing; state clearly that it does not cover fulfilment or customer contact, and record that limit where the executive will see it.
- Handover into business as usual. Change control so that a future application change does not silently break replication; a scheduled test at least annually, ideally before each peak freeze.
Then re-baseline the plan and the BIA documentation with the tested figure - 'eight hours, demonstrated on [date]' rather than 'eight hours, designed for'. The distinction is the whole point of validation.