Deployment Frequency Versus Stability

The most likely reason is not that the team releases too often, but that each release bundles many changes and is verified after deployment rather than before. A two-week release cadence means each release contains roughly two weeks of merged work, so when something breaks it is hard to identify which change caused it, and the rollback discards all of the good changes along with the bad one. The absence of a production-like staging environment and the fact that automated tests run only after deployment mean that defects are discovered by customers rather than by the team. The Thursday evening window also means that problems surface when the people who understand the change are tired and the business is about to open.

To test the hypothesis I would look at the relationship between release size and rollback. I would measure the number of merged changes per release and compare rollback rates for small and large releases. If small releases rarely roll back and large ones often do, the bundling hypothesis is supported. I would also examine the rollbacks themselves: how many were caused by a change that a pre-deployment test could have caught, and how many were caused by environment differences that a staging environment would have caught. That splits the problem into test coverage and environment parity. Finally, I would check whether rollbacks cluster around specific services or authors, which would point to a different cause. The point is to avoid assuming that the cadence itself is the problem; the evidence usually points to batch size and verification timing.

I would treat the cadence as an outcome of capability, not a target to be declared. The first step is to make deployment boring and reversible: automate the deployment steps so they are identical every time, and make rollback a tested, one-command operation rather than an improvised response. Before moving to weekly releases, the team should be able to demonstrate that a rollback takes minutes and has been rehearsed. The second step is to move verification before deployment. That means running the existing automated tests before the release rather than after, and adding tests around the changes that caused recent rollbacks. Before weekly releases, the team should show that the last several rollbacks would have been caught by those tests.

The third step is to create a production-like staging environment, or, if that is too expensive, to use a canary release that sends a small share of traffic to the new version and compares error rates before full rollout. Before weekly releases, the canary should be in routine use. Once those are true, weekly releases are a natural consequence: smaller batches, verified before deployment, with a fast rollback. The move to on-demand deployment should come only after the team has run weekly releases for a quarter with a rollback rate well below the current 18 percent and a restoration time measured in minutes. I would also change the change advisory process so that routine, low-risk changes follow a lighter path, because a heavy approval process for every change discourages the small, frequent releases that reduce risk. The trade-off is upfront investment in automation and testing in exchange for lower risk per release; the two customer-facing rollbacks are evidence for that investment, not against the cadence.