Production deployments counted by job retries
orders-service deploys to staging automatically on every merge to main. Production deploys are manual jobs in the same pipeline, started by a release manager who runs a wrapper script from a laptop. The team counts deployment frequency by counting successful production jobs, but the release manager often re-runs the same job after a failed deploy, so one release can produce two or three records. Lead time is measured from first commit to production job start. Many issues sit in Ready for QA for six to nine days. The team has 14 production deploys in the last 28 days, 3 incidents, and a change failure rate the VP calls too low to be true. Value stream analytics shows 11 days from issue creation to first commit. The team wants better delivery visibility without a new observability vendor and has one engineer for two days.
1. Which delivery metrics are unreliable and why? Define deployment frequency, lead time for changes, and change failure rate correctly.
Read the worked answer
Deployment frequency is unreliable because it counts job retries as separate deployments. A retry after a failed deploy is another attempt, not another release. Count unique successful production deployments to the production environment, deduplicated by release/tag or deployment ID. Lead time is suspect if it starts at first commit and ends at job start: it mixes code review and QA wait with deploy time, and it ignores the actual production start. DORA lead time for changes is from merged commit to successfully running in production; measure from merge commit to successful production deployment. Change failure rate is too low because incidents are not linked to deployments; it should be the percentage of production deployments that cause an incident, rollback, or hotfix within a defined window, for example 7 days. If you only count incidents that someone remembers to label, the rate is understated. Use one definition per metric and state the window.
2. How would you instrument production deployments and incidents in GitLab without a new vendor?
Read the worked answer
Make production a GitLab environment and deploy through a manual job that declares environment: production; that creates a deployment record with timestamp, SHA, and job. Remove the laptop wrapper script or have it call the GitLab API to create a deployment. Do not count failed or canceled deploy jobs; count successful deployments to the environment. For incidents, use issue type Incident in the same group, label them incident, and link each incident to the deployment or MR that caused it using related issues or the deployment reference. Add a required field in the incident template for the deployment ID or tag. Then compute change failure rate as incidents linked to deployments divided by successful deployments in the window. Use Value Stream Analytics and DORA metrics at group level where available, and a small dashboard or scheduled report for the rest. This uses existing issues, environments, and API, not a new observability vendor.
3. Based on the lead-time breakdown, what should the team improve first, and how should they present it?
Read the worked answer
The biggest delay is not deployment; it is six to nine days in Ready for QA plus 11 days from issue creation to first commit. Improving deploy speed will not fix a 14-20 day lead time. First, measure the QA queue separately: time in Ready for QA, review time, and blocked time. Then reduce batch size and WIP in QA, add a definition of ready, and make QA capacity visible. The 11 days before first commit suggests refinement or priority delay; split issues and start smaller. Present a stacked lead-time chart: issue creation -> first commit (11 days), first commit -> merge, merge -> production deploy. State that deployment frequency is 14 attempts but fewer unique releases, and change failure rate is understated until incidents are linked. Propose one improvement for the next 30 days: cut QA wait by half, with a baseline and target. Avoid broad improve CI/CD language; name the queue.