Shift-Left at CloudDesk
CloudDesk provides a multi-tenant SaaS platform built from 40 microservices owned by 12 product teams. The platform deploys about 25 times a week, but the change failure rate is 28% and mean time to restore service is 4 hours 40 minutes. When an incident occurs, 60% are escalated to a central Operations Bridge of six engineers; product teams hand over to the bridge and are not routinely involved in diagnosis. Developers have no production access, and test automation covers 35% of regression scenarios. About 20% of each sprint is consumed by unplanned work. Post-incident reviews are held only for severity-1 incidents and focus on individual error. Leadership wants faster restoration and fewer failed changes without increasing headcount, and is nervous about giving developers more production responsibility.
1. Apply shift-left and swarming principles to redesign support. What would change in roles, access and escalation, and what controls would you keep?
Read the worked answer
Shift-left means moving knowledge and responsibility for support closer to the team that builds the service. Swarming means assembling the right people to resolve an incident rather than passing it through tiers.
Product teams should take first-line on-call for their own services. The Operations Bridge should handle cross-service coordination, major incidents and out-of-hours triage. Give developers constrained production access: read-only diagnostics first, then break-glass write access with approval, session recording and time limits. Replace sequential escalation with a swarm channel that pulls in the service owner, database specialist or network engineer based on service mapping, not job title.
Keep controls: change enablement for production changes, segregation of duties for break-glass, audit logs and an incident commander for severity-1. The risk is that developers are overloaded or fix production without change control. Mitigate with on-call rotas, error budgets and a rule that emergency changes are still recorded and reviewed. Measure mean time to restore, the percentage of incidents resolved by product teams, escalation rate and repeat incidents. The aim is not to eliminate the bridge but to reserve it for work that genuinely needs cross-team coordination.
2. How should release, deployment and change enablement be organised to reduce the change failure rate? Distinguish release from deployment and explain progressive delivery.
Read the worked answer
Separate release from deployment. Deployment is the technical act of putting a version into an environment; release is making a feature available to users. CloudDesk can deploy continuously behind feature flags and release on business approval.
Reduce the change failure rate with smaller batch sizes, trunk-based development, automated regression raised from 35% toward risk-based coverage, contract tests across the 40 microservices, and progressive delivery. Progressive delivery means canary or blue/green deployment with automated rollback and a small blast radius. Change enablement should treat standard, automated deployments as pre-approved standard changes with pipeline evidence, while high-risk changes such as schema, security or tenant isolation retain review.
The 28% failure rate suggests the problem is change quality and blast radius, not change control itself. Use post-incident and post-implementation review data to find the top failure modes, then fix those tests. Keep release coordination for tenant-impacting changes and regulatory windows. Measure change failure rate, deployment frequency, lead time for changes, mean time to restore and the percentage of deployments delivered by canary. If failure rate falls but deployment frequency collapses, the controls are too heavy.
3. What would you do about the 20% unplanned work, and how would you measure whether the new model is working?
Read the worked answer
Treat unplanned work as a demand and capacity problem, not a discipline problem. First make it visible: categorise interrupt work in the tracker, measure its percentage by team and service, and identify the top sources such as repeat incidents, poor test data, manual requests and fragile environments.
Reduce it at source using problem management for repeat incidents, service request automation, self-service and test-environment reliability. Protect capacity explicitly: reserve a percentage of each sprint for interrupt work based on data, and stop starting new feature work when the reserve is exhausted. Use continual improvement and value stream mapping to remove the largest causes.
Measure unplanned work percentage, interrupt-driven lead time, repeat incident rate, percentage of requests automated, and planned versus actual capacity. Track developer experience and on-call load as well, because a model that burns out product teams will fail. Review monthly with the teams. If unplanned work remains at 20% after source fixes, the cause may be external demand that needs a service-level or product decision, not more heroics.