Bimodal cycle time in site reliability engineering
A nine-person site reliability engineering team handles changes to a payments platform. Work is tracked in a Jira board with columns Backlog, In Progress, Code Review, Test, and Deploy. Over 12 weeks, throughput is steady at 18 items per week, but lead time has two peaks: half the items finish in 2-3 days, and the other half take 15-22 days. The board shows an average of 38 items in progress, with a WIP limit of 5 per person. 'Blocked' is recorded in a text field that is rarely filled. The team holds a 90-minute retrospective every four weeks; actions are assigned but not revisited. A new dashboard shows utilization at 92%, which managers praise. Deployments fail 18% of the time, and failed deployments re-enter at Test. The team wants to improve but argues about whether to add a code review checklist or buy a new CI tool.
1. Which flow signals would you make visible, and why those rather than the utilization dashboard?
Read the worked answer
The bimodal lead time is the key signal. A scatterplot of cycle time by item type and date will show whether the long tail is a class of larger changes, a hidden rework loop from failed deployments, or queueing. Make visible: cycle time scatterplot with 50th, 85th, and 95th percentiles; WIP age, not just count; blocked time and reason codes; flow efficiency; failure demand rate from the 18% deployment failures; queue sizes per stage; and rework loops. Replace utilization as the primary metric; 92% utilization explains long queues, not good performance. Add a cumulative flow diagram to show where work accumulates. The 18% failure rate is a flow signal because failed deployments create failure demand and inflate lead time. Segment by change type and risk. Without segmentation, averages hide the two populations. The team's argument about a checklist versus a CI tool is premature; first show where the delay and rework occur.
2. Design feedback loops at different cadences to turn these signals into decisions.
Read the worked answer
Use three loops with different purposes. Daily, 15 minutes: walk the board, focus on aging items, blockers, and pull decisions, not status reporting. Weekly, 45 minutes: service review with requesters and managers; examine lead-time percentiles, demand mix, expedite use, and policy fit; decide one adjustment. Monthly or quarterly, 90 minutes: operations review; examine trends, experiment results, capability, and system constraints; include the CI tool decision as an experiment, not a purchase debate. Replace the four-week retrospective with a shorter monthly learning review tied to flow data; track actions in the same system and review them at the next loop. Add a post-incident or failed-deployment review as a feedback loop for rework. Each loop needs a decision owner and a stated decision rule. The current retrospective fails because actions are not revisited and are disconnected from flow signals. Keep the daily loop short so it survives.
3. Propose the next experiment with a hypothesis, measures, and a decision rule.
Read the worked answer
Experiment: make blocked work and ready criteria explicit before changing tools. Hypothesis: if we add blocked reason codes and a ready-to-test definition, and cap Test WIP at six, the 85th-percentile lead time will fall by at least 20% in four weeks without reducing throughput. Measures: 50th and 85th percentile cycle time, blocked days per item, Test queue age, deployment failure rate, and throughput. Baseline for two weeks. Run for four weeks; review weekly. Decision rule: adopt if the 85th percentile improves and throughput does not fall; adapt if blocked time falls but lead time does not; abandon if no change. Do not run both checklist and CI tool simultaneously; that confounds the result. The failed-deployment loop is a separate experiment: add a pre-deploy checklist for the top two failure causes and measure failure rate. This is a small, reversible change that tests the system, not a generic retrospective action.