Four Teams, One STAGE-2 Slot
A retail platform has four teams - search, checkout, payments and mobile - sharing one integration environment, STAGE-2, which can host only one release candidate at a time. Each team wants a slot twice per sprint; the booking queue is currently 11 days. Teams respond by batching: checkout bundles 9 to 12 changes per slot. When an integration test fails, identifying the offending change takes 2.5 days on average, and 38% of slots end with the environment in an unknown state that needs a full rebuild, which costs one engineer six hours. Deployment frequency is three per month and the change failure rate is 27%. Management proposes buying a larger server so two release candidates can run in parallel, and asking teams to test more carefully before booking.
1. Is STAGE-2 the real constraint? Evaluate the bigger-server proposal using queue and batch reasoning.
Read the worked answer
First separate process time from wait time. A test run in STAGE-2 probably takes hours; the queue is 11 days. When waiting dwarfs processing, you are looking at a scheduling and utilisation policy, not a shortage of machines.
Doubling capacity would roughly halve the queue if demand and variability stayed constant - but demand will not stay constant. Cheaper slots invite more bookings, and teams that currently batch nine changes will simply book more often with the same batch size, because booking remains the expensive, scarce act. Meanwhile the two problems that actually hurt - 2.5 days to find the offending change and 38% of slots left dirty - are unaffected by server size.
Batch size is the dominant variable. Nine to twelve changes per slot guarantees that failure localisation is a search problem, and guarantees that a failure wastes an entire scarce slot, which raises the stakes and encourages even more careful, even later testing.
"Test more carefully" is a request for individual vigilance in a system that rewards batching. It will not change the numbers.
2. What changes would reduce the 2.5-day localisation time and the 38% dirty-environment rate? Explain the mechanism for each.
Read the worked answer
Shrink the batch. Cap a slot at one or two changes. Localisation time is roughly proportional to the number of candidate changes, so this is the single highest-leverage move. It only works if slots stop being scarce, so it must be paired with the next item.
Make environments cheap and reproducible. Define STAGE-2 as code and create per-team, disposable instances. When environment creation takes minutes, the queue disappears and the booking politics disappear with it.
Guarantee a known start state. Provision from the same definition for every run, run a smoke test at slot entry, and tear down at exit. That is what converts the 38% dirty rate into a rounding error - the environment can no longer drift because it is recreated, not reused.
Make failures attributable. Tag every deployment with its commit, put deployment markers into monitoring, and automate bisection across the candidate changes. A failing test should name a change, not start an investigation.
Replace some integration tests with contract tests so most changes never need the shared environment at all.
3. Which measures would you instrument to prove the change worked, and which would you expect to move first?
Read the worked answer
Instrument the flow, not the utilisation.
- Lead time for changes (commit to production) per team, with the waiting portion broken out.
- STAGE-2 queue time and the batch size distribution per slot.
- Localisation time for failed integration runs.
- Share of slots ending in an unknown state and rebuild hours consumed.
- Change failure rate and MTTR, to confirm nothing was traded away.
- Deployment frequency, last, because it is an outcome rather than a lever.
Expected sequence: queue time and batch size move within days to weeks, because they respond directly to the environment and booking changes. Localisation time follows as batches shrink. Change failure rate improves over the following weeks as smaller changes reach production. Deployment frequency moves last and most slowly, because it is the aggregate result of all the others.
Two traps to avoid: reporting averages without the distribution, since a few 12-change batches can hide the improvement; and treating high STAGE-2 utilisation as a good sign, when it is precisely what created the queue.