Harborline Payments: The Recovery Plan That Only Exists on Paper

Neither extreme. Do not run a test you already know will fail just to hit a date, and do not defer testing until everything is fixed.

What the regulator is assessing is evidence that the capability is exercised and that findings drive remediation. A test producing a nine-hour result plus a credible, funded remediation plan is better evidence than a scoped rehearsal engineered to succeed. But the identity service single point of failure is a known defect, not a test finding - running a full test before it is fixed burns two weekends and $180,000 to prove something already documented.

Recommended sequence:

1. Fix the three known blockers first - identity service availability at the DR site, DNS support terms or a secondary provider, and runbook currency. These are weeks of work, not quarters, and they are the difference between a test and a stunt.
2. Run a full-scale test of the Tier-1 authorization path with the regulator's observers present, on a date you can defend, with the RTO clock defined in advance: declaration to restored transaction processing, measured and evidenced.
3. Produce a written remediation plan with dated actions and owners. That plan, not the test transcript, is the artefact the regulator will actually assess.

On the $180,000: frame it as the cost of evidence rather than the cost of a test, and compare it with four days of a peer institution's disruption. Also confirm whether burst capacity can be reserved at a lower rate than on-demand before submitting the figure.

The blockers, roughly in order of how much time they add to the RTO:

1. Identity service single point of failure. A hard dependency existing only in the primary site means the true RTO is 'identity recovery time plus four hours,' whatever the documentation claims. Extend the identity service to the DR site, or run a local authentication cache with a documented staleness limit. Until this is resolved, the four-hour RTO is an aspiration.
2. DNS. A supplier guaranteeing 24-hour response cannot support a four-hour RTO. Contract for faster response with penalties, add a secondary provider, or bring authoritative DNS in-house. This is a procurement decision with an obvious price attached, which makes it easy to approve.
3. Runbooks and knowledge. Three of nine authors leaving is not reversible. Make runbooks owned by role rather than by person, and validate them in a workshop with current staff so they are tested by the people who will use them under pressure.
4. Configuration drift. 40% batch failure means the DR environment is not a replica. Automate configuration management and add a pre-test validation step comparing DR against a known-good baseline.
5. Batch scheduling and upstream data feeds. Map the critical chain, not just the application list.

Present it as a dependency register ranked by 'hours added to RTO,' each with an owner and a cost. That converts an infrastructure complaint into a funding decision the executive can actually make, and it makes the trade-off between spending now and accepting a 9-hour recovery explicit.

Lead with the result. If you present good news first and bury the failure, every subsequent number becomes suspect.

Structure:

1. State it plainly: a partial failover of Tier-1 services took 9 hours 20 minutes against a 4-hour RTO, and 40% of batch jobs failed on missing configuration. Do not soften it as a 'partial success.'
2. Attribute the time. Break the 9h20m into detection, decision, identity recovery, DNS propagation and missing configuration. A time-attributed breakdown converts a failure into a list of fixable items - and shows you understand your own environment.
3. Separate known defects from test findings. The identity dependency was a known architectural risk; the batch configuration failures were a genuine discovery. Distinguishing them demonstrates the test had value and that you were not blindsided.
4. Present remediation with dates, owners and cost, including the $180,000 and the DNS and identity fixes, plus a target RTO for the next test.
5. State the current residual position bluntly: until remediation completes, Tier-1 services should be described as having an RTO of about ten hours, not four. Recommend the board minute the corrected figure.

Why this works: the regulator is testing whether you know your own recovery capability. A documented 9h20m with a funded plan reads as control; an undocumented four-hour claim reads as a finding waiting to happen. Be prepared for the harder follow-up - whether the four-hour RTO was ever achievable and whether the business impact analysis still supports it.