Config Drift and a 45-Minute Checkout Outage
ShopStream deploys to production about 20 times a week through a CI/CD pipeline. There are four environments: dev, test, staging and production. Last week a payment-service timeout value was present in staging but missing in production. The deployment succeeded, no smoke test covered payment authorisation, and checkout failed for 45 minutes. Rollback is a manual runbook step taking about 40 minutes, performed by the on-call engineer, who is the only person who has run it in the last year. Staging uses a different database version from production and has feature flags permanently enabled. The pipeline runs unit tests and a 20-minute API suite in test, but there is no post-deployment verification. The team wants to move to multiple daily production deploys.
1. Identify the highest-value changes to the deployment pipeline and justify the order.
Read the worked answer
Order by risk reduction and speed of detection. The incident was a configuration mismatch, not a code defect, so first make configuration part of the deployed artifact: config as code, versioned per environment, with schema validation that fails the pipeline when a required value is missing.
Second, add post-deployment verification: a smoke test and a synthetic transaction that exercise payment authorisation in production. This detects the failure in minutes rather than through customer reports.
Third, reduce recovery time. Automate rollback or use blue-green switching so recovery does not depend on one on-call engineer and a 40-minute runbook. Then introduce canary releases to limit blast radius. Feature flags can decouple deploy from release once flag handling is disciplined.
Only after these controls exist should deploy frequency increase. More deploys without them multiply exposure to the same failure mode.
2. How should the team test the deployment process itself, not just the application?
Read the worked answer
Treat deployment scripts, configuration and environment definitions as testable artifacts. Add pipeline checks for config drift and environment parity, post-deployment smoke tests per environment, and synthetic monitoring from outside the system.
Rehearse failure: run rollback drills, inject dependency failures, and confirm that alerts fire and the on-call engineer can act. Verify that the rollback restores both application and configuration state, not just the previous container image. Measure lead time, change failure rate and mean time to recovery before and after.
A green application test suite does not prove the deployment worked. The questions are whether the right artifact reached the right environment with the right configuration, whether customers can complete a critical journey, and whether the team can recover quickly when they cannot.
3. Staging has feature flags permanently enabled. What risk does this create and how should it be addressed?
Read the worked answer
Staging is not representative. It exercises only the 'on' path, while production may default to 'off', so untested code paths reach customers. Flag interactions also multiply: two flags give four combinations, and a handful give far more than any team tests.
Address it by managing flag configuration as code per environment and testing both states for critical journeys. Use a small pairwise matrix of flag combinations rather than exhaustive coverage, prioritised by customer impact. Include the production default state in the deployment smoke test.
Add ownership and expiry dates so stale flags are removed rather than accumulating. Ephemeral preview environments per branch can test flag combinations before merge. Do not accept 'staging is close enough' as the basis for multiple daily production deploys.