The Overnight Settlement Patch
Your team runs a nightly settlement pipeline that feeds 14 regulatory reports. A vendor patch fixes a rounding defect that has caused 0.7% of end-of-day balances to differ by one cent, but the patch changes the ordering of three aggregation steps. Testing passed on 30 days of historical data; the error rate in test was 0.02%. The production window is three hours, and a failed run cannot be rolled back after 06:00 because downstream files are locked. A regulator expects the corrected reports in nine business days. The vendor will not support the old version after that date. Your operations lead wants to ship tonight; your control lead wants to wait for a second test cycle that would consume six of the nine days.
1. What is your decision for tonight, and what condition would most change it? Show the risk comparison you used.
Read the worked answer
Decision: Do not ship the patch tonight. Run it in shadow mode for two consecutive production cycles, reconcile all 14 reports, and make a go/no-go decision on day 3.
Reasoning:
- The 0.02% test error looks small, but it was measured on 30 historical days, not on the three-step ordering change in production. The failure mode is severe because downstream files lock at 06:00 and cannot be rolled back.
- The regulator clock is nine business days. A six-day second test cycle leaves three days, which is too little recovery time if the test uncovers a defect. Two shadow nights cost two days and leave seven.
- Compare expected harm, not headline error rates: a 0.7% one-cent defect in live reports is a known, bounded reconciliation problem; an unobserved ordering error can corrupt all 14 reports simultaneously.
Condition that changes the decision: If the shadow runs match all 14 reports exactly, vendor support confirms the old version remains available for the full nine days, and a tested rollback path exists before the 06:00 lock, ship on day 3. If any of those fail, escalate for manual reconciliation rather than deploy blind.
2. If you are overruled and must ship tonight, define stop criteria and the first-hour checks.
Read the worked answer
Stop criteria for the first production run:
- Any of the 14 report control totals differs by more than one cent per account, or more than 5 unexplained account-level breaks.
- Row counts differ by more than 0.1% from the shadow baseline, or any checksum fails.
- Runtime passes 2 hours 15 minutes, leaving less than 45 minutes before the 06:00 lock.
- Any downstream file lock is reached without a completed reconciliation.
First-hour checks:
- Recompute a sample of 100 accounts independently from the source ledger and compare to report output.
- Validate aggregation order by re-running one report segment with the old and new ordering and explaining every difference.
- Confirm the three most regulator-sensitive reports reconcile before allowing the remaining 11 to proceed.
Decision point: Set a hard 05:15 go/no-go. If checks are not clean, hold downstream files and invoke the manual exception process; do not let the batch complete into locked files.