Northwind Streaming's Three-Day Lead Time
Northwind Streaming deploys 30–40 times a day across 60 microservices owned by nine teams. A shared Jenkins pipeline pushes builds through six environments, including a staging environment that drifted from production after a manual hotfix three months ago. Production uses a 5% canary for 30 minutes, but several services share a database schema, and schema changes are applied by a separate DBA team through a ticket queue. Deployment frequency is high, yet lead time from code commit to production averages three days and 18% of production deployments cause customer-visible incidents. The CMDB lists 2,000 configuration items, but 30% have not been verified in the last year. Feature flags are used inconsistently; two teams release features by deploying to all users. The platform team wants to standardise deployment without slowing teams down.
1. Map the deployment value stream from code commit to live service. Identify where deployment management adds value and where waits, handoffs or rework are inflating lead time.
Read the worked answer
Value stream steps: plan and prioritise → develop and commit → build and unit test → deploy to development/test → deploy to staging → security and quality gates → deploy to production → verify → release (flag) → operate and monitor. Where deployment management adds value: planning and scheduling promotion across environments, coordinating the database change with the service deployment, verifying that what reached production matches the intended version and configuration, and handing over to operations. Waste and handoffs:
- The DBA ticket queue is a queue, not a deployment step. It adds wait time and serialises teams. A three-day lead time against 30–40 daily deployments suggests most of the delay is waiting, not executing.
- Six environments with drift create rework and false confidence; staging no longer predicts production.
- Inconsistent feature flags mean two teams conflate deployment with release, so the value stream has no single release control point.
- Unverified configuration items undermine impact assessment and rollback decisions. Improvement focus: reduce wait states first (self-service database migration with guardrails, environment as code, standard change pre-approval for low-risk deployments); then raise the canary's ability to detect failure. The constraint is coordination and environment fidelity, not pipeline execution speed.
2. Recommend deployment patterns and environment/tooling changes. Justify each against the shared-schema risk, the weak canary, the CMDB gap and inconsistent feature flags.
Read the worked answer
Pattern choice: canary is appropriate for customer-facing stateless services, but a 30-minute, 5% canary is too short and too narrow to detect slow-burning errors such as failures that emerge after cache expiry. Extend duration and add automated analysis of error rate, latency and business KPIs, with automatic promote or abort. Blue-green suits stateless services needing instant rollback, but it does not solve the shared database problem and can double cost if not managed. Shared schema: adopt expand-and-contract migrations - deploy backward-compatible schema changes first, keep old and new code able to run, then remove old structures in a later deployment. This decouples DBA changes from application deployment and makes rollback realistic. Environments and tooling: manage environment configuration as code and rebuild staging from the same definitions as production; eliminate manual hotfixes or record them as code. Offer ephemeral preview environments for teams while keeping one production-like staging gate for integration. CMDB: reconcile configuration items automatically from deployment and discovery sources, and make deployment flag or fail when the target CI is unverified. Deployment records become the authoritative change history. Feature flags: one release policy, named flag owners, expiry dates and an audit of flag state. This separates deploy from release and lets teams deploy without exposing features prematurely. Standardise the pipeline with templates rather than one mandatory path, so controls are consistent while teams keep autonomy.
3. Define a small set of metrics for this value stream and a 90-day improvement sequence. Explain how you would prevent the numbers from being gamed, especially around deployments that partly work.
Read the worked answer
Metrics: (1) Deployment frequency per service - throughput. (2) Change failure rate: percentage of production deployments requiring hotfix or rollback, or causing a customer-visible incident within seven days. (3) Lead time from commit to production, split into execution time and wait time to expose queues. (4) MTTR for deployment-caused incidents. (5) Environment fidelity: percentage of deployments where staging and production configuration match, and percentage of verified configuration items. 90-day sequence:
- Weeks 1–2: baseline all five from pipeline, incident and CMDB data; agree definitions with teams so figures are comparable.
- Weeks 3–6: attack wait time with database migration self-service, environment as code and pre-approved standard changes; target lead time under one day.
- Weeks 7–10: strengthen the canary with automated analysis, run a rollback game day, and fix the top two failure causes from incident review.
- Weeks 11–13: review change failure rate and MTTR against baseline and publish a blameless review. Avoiding gaming: measure outcomes such as customer-visible incidents and rollback rate alongside speed; sample deployments for verification; never rank teams on a single metric. Count partial deployments as failures with a severity, otherwise high deployment frequency hides damage. Tie improvements to the value stream constraint, not to tool adoption.