Meridian Retail Bank's Big-Bang Temptation
Meridian Retail Bank runs its teller and account-opening application in 850 branches across three countries. A quarterly release is planned for a Saturday night inside a 01:00–05:00 mainframe batch window. The deployment team proposes a big-bang deployment to every branch, arguing that a phased rollout would stretch over five weekends and consume scarce mainframe access. Rollback scripts exist for application binaries but not for the accompanying database migration. Two of the last four releases produced configuration mismatches on card-reader firmware at 37 branches, causing 90-minute outages. The service desk has 2,400 teller users and no out-of-hours deployment liaison. A regulatory change freeze starts in 21 days. The release includes a customer-notification feature that will be switched off by a feature flag for the first month.
1. Evaluate the big-bang proposal against a phased deployment. Recommend an approach, justify it using the constraints given, and describe the structure of the deployment you would run.
Read the worked answer
Recommendation: risk-based phased deployment, not big-bang. Reasoning:
- Rollback gap: the database migration cannot be reversed with existing scripts. A big-bang failure would affect 850 branches at once with no clean rollback, so exposure must be capped.
- Evidence of configuration fragility: firmware/config mismatches at 37 branches in two of the last four releases show that the branch environment, not just the binaries, is a failure source. A pilot surfaces this cheaply.
- Window maths: five full weekends is not the only option. Three waves (40 pilot branches, then 400, then 410) fit the 01:00–05:00 batch window if each wave includes verification and a 24-hour soak before the next.
- Regulatory freeze: 21 days allows a pilot plus one wave; the remainder should be planned after the freeze rather than compressed. Structure: select cohorts by branch profile (firmware version, transaction volume); define entry and exit criteria per wave (batch completion, error rate within threshold, configuration verification signed off); hold a decision point before each wave. The rollback plan must distinguish application rollback from database rollback. If the migration is one-way, obtain a tested forward-fix and delay deployment until it exists, or choose a backward-compatible migration approach. The feature flag means the notification capability is deployed but not released; release remains a separate decision.
2. At 60% of branches the deployment reports success, but monitoring shows error rates four times normal in 12 branches. The database migration is irreversible. What should the deployment manager do, in what order, and who decides on rollback versus continuation?
Read the worked answer
First action: halt the remaining waves. A fourfold error rate in 12 of the branches deployed is a pattern, not noise. Continuing increases the blast radius while the team is still diagnosing. Contain and decide:
- Freeze further deployment. Keep the deployment manager as single coordinator and open a major incident if customer impact is material.
- Determine whether the 12 branches share a common attribute (firmware version, batch timing, configuration drift). That determines whether the fix is local or systemic.
- Rollback assessment: because the database migration is irreversible, application rollback alone may leave data inconsistency. Compare (a) application rollback plus data reconciliation, (b) forward fix, (c) isolating affected branches. The deployment manager recommends; change enablement authorises rollback or emergency change; the incident manager owns service restoration. This separation prevents the person under pressure from self-approving. Verify and communicate: confirm actual transaction success rather than process completion, reconcile batches, update the service desk with known error and workaround, and give branch operations one version of the status. Record: capture which branches, which version, which configuration and timing. This feeds the post-implementation review and the next wave's entry criteria. Do not resume until the cause is understood and exit criteria are met.
3. The deployment manager sends the service desk a notice saying 'deployment complete' while the customer-notification feature remains behind a disabled flag. Explain the distinction that matters here and list what must happen before the deployment can be closed and handed over.
Read the worked answer
Deployment is the movement of a component into a live environment; release is making it available for use. Deploying code with the feature flag off means the component exists in production but the business outcome is deliberately withheld. Calling that 'complete' without qualification creates three risks: users report the feature as broken, support material does not match visible behaviour, and the organisation cannot distinguish deployment success from release success. Before closure and handover:
- Verify that the deployed item is the expected version and configuration, and record the flag state with an owner and an activation date.
- Update documentation, known errors, workarounds and support scripts, and confirm the service desk can access them.
- Update CMDB and asset records for changed components, including firmware and configuration items.
- Confirm operational acceptance: monitoring thresholds, batch schedules, and the backup and restore implications of the migration.
- Make the release decision separately, by the accountable business owner, and communicate it to branches. The deployment manager should state exactly what was deployed and what was not enabled. If the release decision is delayed, the deployment can still be a success while the release is pending - not the reverse.