Mobile Banking Release Resilience
A bank’s mobile banking app serves 4 million users. Releases are weekly, but 60% of incidents are caused by configuration drift between development, test, and production. Mean time to restore (MTTR) is 2 hours, and the change failure rate is 15%. Development and operations are separate; operations approves changes, and developers are not on call. A recent outage took 6 hours to diagnose because logs were spread across three tools. The bank wants daily releases and 99.95% availability, but risk and compliance require change traceability. Post-incident reviews focus on finding someone to blame, and staff fear change. There are no error budgets. The CIO has heard about DevOps and high-velocity IT but worries that faster releases will increase risk. The bank has strong automated testing for functional requirements but no environment parity or observability standards.
1. Assess the organization’s current operating model against high-velocity IT principles. Identify the biggest constraints to daily releases and 99.95% availability.
Read the worked answer
HVIT principles relevant here: lean flow, fast feedback, resilience, automation, and a culture of learning.
Constraints:
- Configuration drift: environments differ, so a change that passes test can fail in production. This directly causes 60% of incidents and undermines daily releases.
- Separation of dev and ops with approval handoffs: operations approves changes but developers are not on call. This creates slow feedback and a blame boundary; MTTR stays at 2 hours because the people who can fix code are not in the incident.
- Siloed monitoring: logs in three tools increase diagnosis time (6-hour outage). Without observability, fast restoration is impossible.
- No error budgets: risk is managed by opinion and approval, not by measured reliability. This makes daily releases feel unsafe.
- Blame-oriented post-incident reviews: staff hide information, so learning is slow and incidents recur.
Reasoning: The biggest constraint is not release frequency itself but the lack of environment parity and fast feedback. Daily releases will increase risk unless drift, observability, and on-call ownership are addressed first.
2. Propose a target operating model and practices that improve resilience and flow while preserving change traceability for compliance. Explain the reasoning.
Read the worked answer
Target operating model: cross-functional product teams own the app end to end, including on-call. A platform team provides CI/CD, infrastructure as code, and observability. Change enablement shifts from manual approval of every release to automated, risk-based controls with traceable records.
Practices:
- Infrastructure as code and environment parity: build dev, test, and production from the same templates. This removes configuration drift and makes test results credible.
- CI/CD with automated change records: every deployment produces an immutable audit trail (who, what, when, approvals, test evidence). This satisfies compliance without slowing flow.
- Progressive delivery: canary releases and feature flags limit blast radius. If a canary fails, rollback is fast, protecting availability.
- Observability and error budgets: unified logs, metrics, and traces; define an error budget for 99.95%. Release pace is governed by budget burn, not by fear.
- Blameless post-incident reviews: focus on system conditions and learning. This improves resilience and encourages reporting.
Reasoning: Traceability and speed are not opposites if evidence is generated automatically. The model replaces manual control with automated guardrails, which is consistent with HVIT’s resilient operating model.
3. What cultural and measurement changes are necessary, and how would you sequence them?
Read the worked answer
Cultural changes: move from blame to blameless learning; make developers share on-call; reward incident transparency and fast restoration; give teams autonomy within guardrails. Leaders must visibly support experimentation and rollback without punishment.
Measurement changes: track deployment frequency, lead time for changes, change failure rate, MTTR, availability against the 99.95% target, and error-budget burn. Add a learning measure: percentage of incidents with completed blameless reviews and actions. Avoid using incident counts to rank individuals.
Sequencing:
- Stabilize visibility: unify logging and observability, start blameless reviews, and define the error budget. This builds trust and data.
- Remove drift: implement infrastructure as code and environment parity; automate change records.
- Increase flow: introduce CI/CD and progressive delivery; move to daily releases for low-risk changes first.
- Embed culture: extend on-call, publish error budgets, and review outcomes in service reviews.
Reasoning: Culture and measurement reinforce each other. If you push daily releases before observability and blameless learning, incidents will reinforce fear. Sequence from visibility to automation to pace.